Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why google is not visiting any website from past 10 days
Hi, I observed why Google is not visiting www.SubhaVastu.com from past 10 days, later I checked thoroughly, not only for my site, google stopped visiting all websites from past 10/12 days. Is Google releasing any new updates to the crawler? Any new system is releasing soon. I am expecting Google updated their crawler by this Sunday night and it may visit as usual to all sites from 12-midnight pacific time. Has anyone observed it, any information regarding on this Google step. Thanks.
Algorithm Updates | | SubhaVaastu0 -
Googles Search Intent – Plural & Singular KW’s
This is more of a ‘gripe’ than a question, but I would love to hear people’s views. Typically, when you search for a product using the singular and plural versions of the keyword Google delivers different SERPs. As an example, ‘leather handbag’ and ‘leather handbags’ return different results, but surely the search intent is exactly the same? You’d have thought Google was now clever enough to work this out. We tend to optimise our webpages for both the plural and singular variations of the KW’s, but see a mixed bag of results when analysing rankings. Is Google trying to force us to create a unique webpage for the singular version, and another unique webpage for the plural version? This would confuse the visitor, and make no sense.. the search intent is the same! How do you combat this problem? Many thanks in advance. Lee.
Algorithm Updates | | Webpresence0 -
Google live blog schema
Live blog markup was rolled out to a selected group of publishers last year 2015. I'm trying to find out whether it has been released to other publishers yet (we are a news site). Not seeing any updates about it anywhere and wondering how I can find out if/when it will be available. This is the latest I can see which is not much help https://developers.google.com/search/pilot/open/live-blogs. Any insider info would be very much appreciated.
Algorithm Updates | | hjsand2 -
What are the top tips for winning on Google, Bing, and Yahoo?
We just launched a new site that is starting to be indexed well in Google, but Bing and Yahoo are lagging a bit. I understand that the search engines are different algorithms and will take different lengths of time to index, rank, etc. What I'm curious about is are there any other tips / advice / things to keep in mind when trying to rank on the different search engines? Thanks!!
Algorithm Updates | | Emily_A0 -
How long does google take to re-ranking pages in results?
I mean when google dance, the pages in results go up and down frequency every minue, but finally your page will rank in any position in google, what is the time when you get another position in google
Algorithm Updates | | engtamous0 -
My Website No Longer Appears in Mobile Google Search but Does in Desktop...Why Is This?
For a long time my website has appeared in both desktop and mobile search in Google. Yet recently it has stopped appearing in mobile yet still on desktop. Any ideas why this is happening and how to rectify it please? Many Thanks.
Algorithm Updates | | WSIDW0 -
Would Google Remove Pages for Inactivity?
Hi, I've been watching the Total Indexed number for 4 domains that I work with for the last few months. In Google Webmaster Tools three of them were holding steady up until August-September, when suddenly they started declining by hundreds of thousands of URLs a week. I've asked my IT department and they say they haven't done anything technically different in the last few months that would affect indexation. I've also searched on google and on search marketing blogs to see if anyone else has experience this to no avail. As you can see in the image, the "Not Selected" pages have not increased so it appears this is not due to duplicate content (of which we have a lot). However, the "Ever Crawled" number is increasing. The only reasonable answer that I can conclude is that Google is now de-indexing inactive URLs? Anyone have a better answer? yIYDm.jpg
Algorithm Updates | | OfficeFurn0 -
Effect of new Google SSL policy on our Analytics - AACK!
So I went to look at our keyword reports in GA today and our most popular keyword was "(not provided)". It now accounts for 10% of our referred visits. Unfortunately, it also has a 125% avg order value compared to the rest of our site. This is a really annoying policy that Google has implemented and will clearly have an effect on our ability to effectively market our site.
Algorithm Updates | | IanTheScot0