Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Custom Search Engine: Good Idea?
I created a Google Custom Search Engine for our site, but I"m not sure implementing it is a good idea. When I tested it with the public URL, I noticed that ads show up on the search engine that could potentially move visitors away from our site to our competitors. Has anyone had success with implementing a Google Custom Search Engine? Do the pros outweigh the cons? Thanks, Ruben
Algorithm Updates | | KempRugeLawGroup0 -
Advice regarding latest Google Algo Update please, if possible...
Hi there I wonder if anyone can advise on this. Since the latest google update on 1st Sept or whenever it rolled out, we noticed an initial spike in hits on our site, which was great. However now we are noticing levels going back to where they were and less people visiting the site. It also seems to be very sporadic. So we have a period of say a couple of hours with no one on the site, then suddenly loads visiting. We have also noticed a big dip in enquiries, despite the site having roughly the same amount of visitors. All our stats on Moz, Webmaster Tools, Ahrefs, Serpfox and various other rank trackers are showing that we have had an increase in visibility on our tracked keywords. There is a definite spike on all, but where is our traffic and where are our enquiries? Usually we are able to work out where the problem is when updates occur but with this we have no idea. We are utterly baffled. Is this normal? Is this just fluctuations and will settle down? Has anyone else noticed weird things happening? If anyone has any ideas or experience of this then would be most grateful for any advice. Feeling rather desperate at the moment. Many thanks in advance. Clojo
Algorithm Updates | | Clojobobo0 -
Why Google loves MOZ for "Directory Submmission Service" ?
I have just for "directory submission service" in Google.com ( Geo Location USA ). I got two results from moz community for same thread. Does Google don't understand 301 redirect from seomoz.org to moz.com ? What about Domain Clustering ? PFA: SERP Screenshot kn8evtt.png
Algorithm Updates | | SanketPatel0 -
Google Authorship and Hobby Blog
I hope that someone can help me come up with the best option. Please forgive my ignorance on this issue. I have a hobby blog and up until now I have not wanted to associate it with my real name. It is a menswear blog about classic American style. I was afraid that it may be a hindrance if I was ever looking for a more conservative career than SEO. I am now reconsidering this and thinking that claiming it may be of more help than harm. Which brings me to Google Authorship. My dilemma and misunderstanding stems from the fact that I have mutliple Gmail accounts. I am guessing that some of the newer accounts have a G+ associated with them. So my question is do I use the email that is associated with my blog or my main gmail that I use personally? If I do use the gmail associated with the blog will it then become my default Google plus profile? Any insight would be helpful. Thanks in advance. If any of you are interested the hobby blog is Oxford Cloth Button Down.
Algorithm Updates | | JerrodDavid0 -
Google doesnt index my Google+ Profile
Hey guys! I know it sounds like a novice question, but I have checked ALL THE BOXES THAT TELL GOOGLE TO INDEX MY GOOGLE+ PROFILE. It is Visible for search - 100%. It's been 3 weeks since I opened a Google+ profile and it still hasn't been indexed for its name. Any guesses what's going on? (It's not this name so don't try to google me)
Algorithm Updates | | Yoav_Vilner0 -
Proper Way To Submit A Reconsideration Request To Google
Hello, In previous posts, I was speaking about how we were penalized by Google for unnatural links. Basically 50,000 our of our 58,000 links were coming from 4-5 sites with the same exact anchor text and img alt tags. This obviously was causing our issues. Needless to say, I wen through the complete link profile to determine that all of the links besides this were of natrural origins. My question here is what is the accepted protocol of submitting a reinclusion request; For example, how long should it be? Should I disclose that I was in fact using paid links, and now that I removed (or at least nofollowed) them? I want to make sure that the request as good as it should so I can get our rankings up in a timely manner. Also, how long until the request is typically aknowledged? Thanks
Algorithm Updates | | BestOdds0 -
Google indexing my website's Search Results pages. Should I block this?
After running the SEOmoz crawl test, i have a spreadsheet of 11,000 urls of which 6381 urls are search results pages from our website that have been indexed. I know I've read that /search should be blocked from the engines, but can't seem to find that information at this point. Does anyone have facts behind why they should be blocked? Or not blocked?
Algorithm Updates | | Jenny10 -
Can Google display a diffrent page title?
Hi if I search google UK for the phrase car leasing, google returns my listing as Car Lease Deals However the same search on Yahoo or Bing bring back Contract Hire | Vehicle & Car Leasing Deals | Car Lease Deals this is the real page title. Why would this happen? Thanks Andy
Algorithm Updates | | First-VehicleLeasing0