Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How long for google to de-index old pages on my site?
I launched my redesigned website 4 days ago. I submitted a new site map, as well as submitted it to index in search console (google webmasters). I see that when I google my site, My new open graph settings are coming up correct. Still, a lot of my old site pages are definitely still indexed within google. How long will it take for google to drop off or "de-index" my old pages? Due to the way I restructured my website, a lot of the items are no longer available on my site. This is on purpose. I'm a graphic designer, and with the new change, I removed many old portfolio items, as well as any references to web design since I will no longer offering that service. My site is the following:
Algorithm Updates | | rubennunez
http://studio35design.com0 -
Strange Google SERP Layout: Anyone?
I haven't been able to repeat this, but I just saw a strange Google SERP layout. The screenshot is attached below. Has anyone else seen something like this? I kind of rubbed my eyes and wondered 'if it was kicking in'. ca2JqaP
Algorithm Updates | | Travis_Bailey0 -
Is it me or Google?
Hi All, I'm new here so take it easy on me.. OK, basically i have had a SEO company for about 3 years, they did a wonderful job, for the last 12 months or so i have been in top 1-3 positions for pretty much every keyword i wanted... On Jan 17th, that all changed, suddenly google doesnt like something about my site... for the sake of this questions lets focus purely on the keyword "CCTV", i use to be 1st or 2nd, it varied... Since Jan 17th i am all over the place, today alone, i was 9th this morning, then 13th, then 22nd... I am working on a lot of things my SEO company told me to do, with regards my site, obviously this is going to take time... but my big concern is that google doesnt seem to know where to rank me lol, i mean, at least if they settled on a place i.e 22nd, then i have a stable base to work from... Has anyone seen this kind of thing before, and can i expect at somepoint google decides to simply remove me? Any advice welcome. regards James
Algorithm Updates | | isntworkdull0 -
Why does Google Alerts call my website a blog?
Our company started a WordPress blog about 14 years ago. It has since added a third-party forum, a user-submitted photo gallery, and a huge database of searchable products. We also have almost 4000 posts. With all that said, Google Alerts often lists our content under blogs rather than websites. Sometimes it shows up in both? Does anyone know what criteria Google uses for determining the type of content, and how we can signal to them that we are a website?
Algorithm Updates | | TMI.com0 -
Does google index non-public pages ie. members logged in page
hi, I was trying to locate resources on the topics regarding how much the google bot indexes in order to qualify a 'good' site on their engine. For example, our site has many pages that are associated with logged in users and not available to the public until they acquire a login username and password. Although those pages show up in google analytics, they should not be made public in the google index which is what happens. In light of Google trying to qualify a site according to how 'engaged' a user is on the site, I would feel that the activities on those member pages are very important. Can anyone offer suggestions on how Google treats those pages since we are planning to do further SEO optimization of those pages. Thanks
Algorithm Updates | | jumpdates0 -
Did Google Just Punish Organic For Showing Up In Local?
Google pushed the local search results in a prominent spot on a ton of search results. That means (for me) that all first page rankings (including a lot of 1st place rankings) have completely disappeared. Subfolders still rank and homepage still ranks where local search is not included. This is weird. Is this a glitch or is there something we can do about this?
Algorithm Updates | | Discountvc0 -
How often do people use Google Product Search
I was was reading Tom Critchlow's excellent blog on how to rank well for Google Product Search. I'm trying to find out if there are stats on how often people use this feature in Google (since it is not listed on Google's main navigation). I'm working with a customer who has b-2-b products and am trying to determine the value of adjusting his ecommerce pages to appear on Google Product Search.
Algorithm Updates | | EricVallee340 -
Geolocation: Google only crawls from the US
A question was previously asked about geo-location and specifically if Google crawled from other countries. I could not locate the original question but wanted to share the below information. As of earlier this year Google only crawls from US IP addresses: http://www.youtube.com/watch?v=7paVYBgH0Hw
Algorithm Updates | | RyanKent1