Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Https & Google Updated Guidelines
Hi We have https on aspects of the site which users directly interact with, such as login, basket page. But we don't have https across the whole site. In light of Google adding it to their guidelines - is this something we need to put into action? Also same question on the Accessibility point Ensure that your pages are useful for readers with visual impairments, for example, by testing usability with a screen-reader. Are we going to be penalised if these are not added to our site? Thank you
Algorithm Updates | | BeckyKey0 -
Google SERPs changes
Hi I wonder if anyone knew of any changes to the Google SERPs appearance in August 2015? We dropped in over a thousand visits to the homepage on brand so I wanted to find out why. Also, our DA went from 36 to 34 - does Google panda affect domain authority at all? Thank you
Algorithm Updates | | BeckyKey0 -
Does anyone have an idea of the benefits of Google Analytics Premium?
We've been having a discussion about the GA Premium service here in our office, trying to weigh up the pro's and con's... For the majority all it seems you gain access to is more support from google. We're trying to find out if that is the case or if you gain extra information, such as and insight into the search terms who must not be named. Of course i'm talking about the (Not Set) data... This section of data is ever increasing, yes i know we can access certain terms through webmasters but it was so much easier (in the good ol' days) when all the data was under one roof! Any thought opinions or even more questions would be greatly appreciated, i look forward to your responses. Anthony
Algorithm Updates | | Kal-SEO0 -
Would Google Remove Pages for Inactivity?
Hi, I've been watching the Total Indexed number for 4 domains that I work with for the last few months. In Google Webmaster Tools three of them were holding steady up until August-September, when suddenly they started declining by hundreds of thousands of URLs a week. I've asked my IT department and they say they haven't done anything technically different in the last few months that would affect indexation. I've also searched on google and on search marketing blogs to see if anyone else has experience this to no avail. As you can see in the image, the "Not Selected" pages have not increased so it appears this is not due to duplicate content (of which we have a lot). However, the "Ever Crawled" number is increasing. The only reasonable answer that I can conclude is that Google is now de-indexing inactive URLs? Anyone have a better answer? yIYDm.jpg
Algorithm Updates | | OfficeFurn0 -
Google Multiple Results
With Google's penchant for listing at times many results - one on top of the other - from the same domain, is it now advisable to not worry about having multiple pages in the same site targeting the same or very similar keywords? Is this (keyword/page internal competition) one less thing that I have to worry about or worry about less or what? Thanks! Best... Jane
Algorithm Updates | | 945010 -
How often do people use Google Product Search
I was was reading Tom Critchlow's excellent blog on how to rank well for Google Product Search. I'm trying to find out if there are stats on how often people use this feature in Google (since it is not listed on Google's main navigation). I'm working with a customer who has b-2-b products and am trying to determine the value of adjusting his ecommerce pages to appear on Google Product Search.
Algorithm Updates | | EricVallee340 -
Strange Refferral URL coming in from Google
Hi, I've been monitoring my referral URL's coming in and today noticed they had changed. Previously when I clicked one it would be the google search result page - however now they all seem to be like this: http://www.google.co.uk/url?sa=t&source=web&cd=7&sqi=2&ved=0CHEQFjAG&url=http://www.mysite.com&rct=j&q=my%20keyword&ei=Bvc3TrbgB5G0hAfvqoSvAg&usg=AFQjCNFONDCPJDl3d2PYceYvale_cL7s4Q All these URL's immediately redirect to my website pages. Do you know what they are - they seem to be tracking URL's of some sort I am thinking?? Are they trying to analyse my site with respect to certain keywords?? Thanks
Algorithm Updates | | James770