Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Traffic cut-off since Google core update
Hi all, I am the webmaster of www.chepicap.com/en (Cryptocurrency news), and since the 3rd of june (Google core algorithm update) we got the hammer from Google. Organic traffic dropped with 90%+ overnight. We are still in the dark whether we can do to improve the current situation. Does someone have suggestions regarding this issue?
Algorithm Updates | | NielsDE0 -
What we can understand from decrease in % of New Sessions year to year in Google analytics.
Hi, As per Google analytics, we are loosing around 15% of new sessions every year comparing to the previous. But the sessions are increasing. Good that sessions increased; but new sessions are getting reduced. What exactly we can understand from this scenario? Thanks
Algorithm Updates | | vtmoz0 -
Google Search Subsections
Hi! I want to know how can I put the URL from a page like that: http://i.imgur.com/qK1NLjq.png?1 I mean: "www.calafate.com › El Chaltén" Is it possible? Thanks!!!
Algorithm Updates | | Seomediabros0 -
US domain pages showing up in Google UK SERP
Hi, Our website which was predominantly for UK market was setup with a .com extension and only two years ago other domains were added - US (.us) , IE (.ie), EU (.eu) & AU (.com.au) Last year in July, we noticed that few .us domain urls were showing up in UK SERPs and we realized the sitemap for .us site was incorrectly referring to UK (.com) so we corrected that and the .us domain urls stopped appearing in the SERP. Not sure if this actually fixed the issue or was such coincidental. However in last couple of weeks more than 3 .us domain urls are showing for each brand search made on Google UK and sometimes it replaces the .com results all together. I have double checked the PA for US pages, they are far below the UK ones. Has anyone noticed similar behaviour &/or could anyone please help me troubleshoot this issue? Thanks in advance, R
Algorithm Updates | | RaksG0 -
Why is Google.co.uk showing so many US retailers on search queries?
Over the last couple of years Google seems to be showing more and more irrelevant search results in the field I am based in (dietary supplements) with searches for products coming back with results which highlight US vendors. Given I am in google.co.uk and actually logged in as well, why is Google so far off with their results lately?
Algorithm Updates | | predatornutrition0 -
Which Google Snippet to use? Products or Review
A "product" page that includes "user reviews/ratings" and "comparison prices", which Google snippet do/should you use? Thanks in advance
Algorithm Updates | | righty0 -
Webpage is ranking on google.ie / google.co.uk but not google.com?
One of our site webpage appears to be found in the first few pages on google.ie / google.co.uk but not on google.com. Is there such a thing being penalised on a specific Google domain? Traffic is healthy despite this but I want to rank well for the page in google.com. Any ideas?
Algorithm Updates | | notnem0 -
How do you determine if Google thinks you’re guilty of Key Word Stuffing?
I believe that Google sees our website: www.getyourphotosoncanvas.com/
Algorithm Updates | | rdominey
as being guilty of keyword stuffing. I don’t see any specific tools on SEOmoz
that will evaluate Keyword Density as viewed by Google. Just to be clear; I have not attempted to use KWS as a SEO tactic. I feel that the
content of each page is written for the customer. I do realize that Google may
see it differently. I think that I am being penalized for the use of the words
Photo and Canvas. My domain name is “GetYourPhotosonCanvas.com” The business
name is “Get Your Photos on Canvas”, the website title is “Get Your Photos on Canvas”, and each page of our website talks about Photos and Canvas. I have tried to
vary the use of Photo with Pictures, Photographs, Prints, Digital Images and so
on. It is difficult to vary the word Canvas. I guess according to Google this paragraph
would be guilt of Keyword Stuffing! I have conducted some tests on other SEO site tools and some indicate that KWS is
a problem on my website. If you search a specific page title that contains either Photo or Canvas it does
not rank in the top 200 on Google, although it is #1 or #2 on Yahoo or Bing. Drop those two Keywords and the page shows up
as #2 on Google. I think that is a good indication that we are being penalized
for KWS. Pease take a look at our website and give me your opinion/advice regarding the Key
Word Stuffing Issue. Do I need to rewrite my site content for Google?0