Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How serious is Google about internal linking report? Considers the links from sub-directories too?
Hi community members, It's been clearly said by Google to interlink the important pages across the website and they give top interlinked pages in "Links report". They do consider the links from the sub-directories like example.com/blog, etc. to sum up the internal linking . But we do employ multiple sub directories and link to various pages which may not be that important to rank, example "terms of use" page at footer section. So, obviously these non-important pages might be over linked as per the search console "internal links report". Will this make Google to consider the highest linked pages as most important and they try to give ranking importance to them? How about links from sub directories? Please clarify and share your opinions.
Algorithm Updates | | vtmoz0 -
Google not showing the recent cache info: How to know the last cached version of a page?
Hi, We couldn't able to see the last Google cached version of our homepage after March 29th. Just wondering why this is happening with other websites too. When we make some changes to the website, we will wait to our website indexed and cached, so the changes will have some ranking impact. Now we couldn't able to check if the website got indexed with changes. Is there any other way to check the latest cached version or time of last index? Thanks
Algorithm Updates | | vtmoz0 -
Google Cache
So, when I gain a link I always check to see if the page that is linking is in the Google cache. I've noticed recently that more and more pages are actually not showing up in Google's cache, yet still appear in search results. I did read an article from someone whoo works at Google a few weeks back that there is sometimes an error with the cache and occasionally the cache will not display. This week, my own website isn't showing up in the cache yet I'm still ranking in SERP's. I'm not worried about it, mostly whitehat, but has there been any indication that Google are phasing out the ability to check cache's of websites?
Algorithm Updates | | ThorUK0 -
New Google Update - weird ranking
Hi I wanted to get your thoughts on this keyword ranking. This page - https://www.key.co.uk/en/key/albatross-heavy-duty-office-chairs-24-stone is now ranking for heavy duty office chair 30 stone We don't mention 30 in the content anywhere, apart from the USPs at the top of the page - could this be it?! I don't know how to change this, or I guess Google is still figuring things out and maybe this will drop off? Love to get some thoughts on this! Becky
Algorithm Updates | | BeckyKey0 -
The Google Algo update that happened 1-8 is KILLING my rankings
Does anyone know what happened?? I have a great website, we ranked very highly for a slew of industry keywords, #1 in most of our top-money kws....and our keywords have been in freefall since the update. Help?!
Algorithm Updates | | Sean_Gutermuth0 -
New Zealand rankings - Is Google taking city into account now?
I've just noticed over the last 24 hours that Google New Zealand seems to be taking city into account with the search results - when I search from Christchurch - I get CHCH companies and Auckland results when searching from Auckland. I would like to rank my client nationally - any tips? Has anyone else noticed this? I need to be the top of Google for my keyword throughout the country. Any articles that could help? Kind regards, David
Algorithm Updates | | David_Buckingham0 -
How Do I Make My Google SERP "SiteLinks" more relevant?
I have a shopping website with thousands of products, and the sitelinks that google has chosen for me (for a long time) are random product pages, which makes no sense to me. I do not emphasize those products on the home page, and I have a sitemap that clearly lists the directory of all the categories. I also added a "nofollow" attribute to almost every link on the home page that is not important. These products in the site links seem completely random and there isnt even a sitelink for "about" or any of the footer content! What gives? Also, my sitelinks never updated to the new, better version. Any suggestions?
Algorithm Updates | | cDNAInteractive0 -
When Google crawls and indexes a new page does it show up immediately in Google search - "site;"?
We made changes to a site, including the addition of a new page and corresponding link/text changes to existing pages. The changes are not yet showing up in the Google index (“site:”/cache), but, approximately 24 hours after making the changes, The SERP's for this site jumped up. We obtained a new back link about a couple of weeks ago, but it is not yet showing up in OSE, Webmaster Tools, or other tools. Just wondering if you think the Google SERP changes run ahead of what they actually show us in site: or cache updates. Has Google made a significant SERP “adjustment” recently? Thanks.
Algorithm Updates | | richpalpine0