Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Keywords in Paragraphs: How much do they matter at Google?
Hi all, Generally we care a lot about keywords at headings, title tags, URL, etc. always. But I wonder how much impact they have being in paragraphs. How much do they matter at paragraphs? Thanks
Algorithm Updates | | vtmoz0 -
My Website No Longer Appears in Mobile Google Search but Does in Desktop...Why Is This?
For a long time my website has appeared in both desktop and mobile search in Google. Yet recently it has stopped appearing in mobile yet still on desktop. Any ideas why this is happening and how to rectify it please? Many Thanks.
Algorithm Updates | | WSIDW0 -
Question about Google Algo Change on June 26
I have a client who's Google Organic visits dropped significantly on June 26th. I used a chart overlay called ChartIntelligence. It says that there was an SEOF update on 6/26/2013. Does anyone know what this update (or any other updates) would be? Also, where might I find additional info on this update. I did notice that Moz's algo change tracker listed a multi-week update on June 27, but I'm not sure where to find info on what types of things were impacted by this update. Any info would be helpful.
Algorithm Updates | | TopFloor0 -
How to keep damage low on Google after the change of URL's
Hi Peeps, Hope someone can shed a light on this and show a guidance if possible. We are going to move our sites to shopify and shopify's URL's cannot be customized to match exactly like our current URLs. What steps do I need to take so google knows the URL's are changed. Domain will be the same. Thank you in advanced.
Algorithm Updates | | cemalcebi0 -
How to follow up Google search in a Joomla website
Hello ! I'm facing a stupid issue but cannot solve it. Google Analytics can track the seach into a website. When I try to activate it I need to add some more parameters that I do not know. http://awesomescreenshot.com/03f1bnau8d Has anyone ever tried to configure Google analytics search for a Joomla website ? Tks a lot !
Algorithm Updates | | AymanH0 -
Do you think Google is destroying search?
I've seen garbage in google results for some time now, but it seems to be getting worse. I was just searching for a line of text that was in one of our stories from 2009. I just wanted to check that story and I didn't have a direct link. So I did the search and I found one copy of the story, but it wasn't on our site. I knew that it was on the other site as well as ours, because the writer writes for both publications. What I expected to see was the two results, one above the other, depending on which one had more links or better on-page for the query. What I got didn't really surprise me, but I was annoyed. In #1 position was the other site, That was OK by me, but ours wasn't there at all. I'm almost used to that now (not happy about it and trying to change it, but not doing well at all, even after 18 months of trying) What really made me angry was the garbage results that followed. One site, a wordpress blog, has tag pages and category pages being indexed. I didn't count them all but my guess is about 200 results from this blog, one after the other, most of them tag pages, with the same content on every one of them. Then the tag pages stopped and it started with dated archive pages, dozens of them. There were other sites, some with just one entry, some with dozens of tag pages. After that, porn sites, hundreds of them. I got right to the very end - 100 pages of 10 results per page. That blog seems to have done everything wrong, yet it has interesting stats. It is a PR6, yet Alexa ranks it 25,680,321. It has the same text in every headline. Most of the headlines are very short. It has all of the category and tag and archive pages indexed. There is a link to the designer's website on every page. There is a blogroll on every page, with links out to 50 sites. None of the pages appear to have a description. there are dozens of empty H2 tags and the H1 tag is 80% through the document. Yet google lists all of this stuff in the results. I don't remember the last time I saw 100 pages of results, it hasn't happened in a very long time. Is this something new that google is doing? What about the multiple tag and category pages in results - Is this just a special thing google is doing to upset me or are you seeing it too? I did eventually find my page, but not in that list. I found it by using site:mysite.com in the search box.
Algorithm Updates | | loopyal0 -
Google and Wikipedia
Ok, I love Wikipedia as much as the next guy but the amount of weight that google puts on this site is getting crazy. My search terms that I am going after are "speakers" and "loudspeakers" Can somebody tell me why wikipedia needs the top 8 -10 spots for those terms? is that really a good search result for users of google? More of a rant then a question I know. I just needed to get that off my chest!.
Algorithm Updates | | kevin48030 -
Indexing well in Google but not in Yahoo/Bing - WHY?
Been using SEOMOZ now to analyze and crawl a client's website for a while now. One thing I've noticed is that our client's website is indexing well with Google. a few thousand pages are being indexed. However, when it comes to Yahoo and Bing, the website only has a 100+ pages indexed. We've submitted updated sitemaps to Google and Bing and have been fixing any broken links, and on-page SEO. Content is also good. Here's the website: www.imaginet.com.ph Any suggestions/recommendations are highly appreciated. Thank you!
Algorithm Updates | | TheNorthernOffice790