Do I have a robots.txt problem?
-
I have the little yellow exclamation point under my robots.txt fetch as you can see here- http://imgur.com/wuWdtvO
This version shows no errors or warnings- http://imgur.com/uqbmbug
Under the tester I can currently see the latest version. This site hasn't changed URLs recently, and we haven't made any changes to the robots.txt file for two years. This problem just started in the last month. Should I worry?
-
Today it has a green check mark, and absolutely no changes were made to the website since I asked this question.
-
It could be that your server had a hard time when Google tried to view your robots.txt file that's why it wouldn't be able to fetch it. As long as this issue doesn't prevent Google anymore in the future it's not much to worry about.
-
That would make me feel more confident of a false error being reported. Time to closely monitor the crawl logs, look at server stats, and keep an eye on GWT for a change in the reporting/indexing. I would also go into the GWT forums and post, see if anyone is reporting a similar error these past couple days.
-
I can't post the domain but I know it is accessible.
When I go to the tester it shows the live robots.txt with no problems. I also can look at the server logs and see that it is being crawled, but being crawled less then Bing Crawls. Also the Bing Webmaster Tools is showing no problems.
-
Can you post your domain? Manually checking the robots.txt file would help.
I've checked many of my GWT accounts and I am not showing a sudden robots.txt error. It could be a false error, but I would take anything with the robots.txt file seriously. You'll want to make sure that it is in fact accessible to all the crawlers desired.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's wrong with this robots.txt
Hi. really struggling with the robots.txt file
Technical SEO | | Leonie-Kramer
this is it: User-agent: *
Disallow: /product/ #old sitemap
Disallow: /media/name.xml When testing in w3c.org everything looks good, testing is okay, but when uploading it to the server, Google webmaster tools gives 3 errors. Checked it with my collegue we both don't know what's wrong. Can someone take a look at this and give me the solution.
Thanx in advance! Leonie1 -
'External nofollow' in a robots meta tag? (advertorial links)
I believe this has never worked? It'd be an easy way of preventing any penalties from Google's recent crackdown on paid links via advertorials. When it's not possible to nofollow each external link individually, what are people doing? Nofollowing and/or noindexing the whole page?
Technical SEO | | Alex-Harford0 -
I accidentally blocked Google with Robots.txt. What next?
Last week I uploaded my site and forgot to remove the robots.txt file with this text: User-agent: * Disallow: / I dropped from page 11 on my main keywords to past page 50. I caught it 2-3 days later and have now fixed it. I re-imported my site map with Webmaster Tools and I also did a Fetch as Google through Webmaster Tools. I tweeted out my URL to hopefully get Google to crawl it faster too. Webmaster Tools no longer says that the site is experiencing outages, but when I look at my blocked URLs it still says 249 are blocked. That's actually gone up since I made the fix. In the Google search results, it still no longer has my page title and the description still says "A description for this result is not available because of this site's robots.txt – learn more." How will this affect me long-term? When will I recover my rankings? Is there anything else I can do? Thanks for your input! www.decalsforthewall.com
Technical SEO | | Webmaster1230 -
Roger has detected a problem
SEOMOZ says Roger has detected a problem: We have detected that the domain www.romancebookstore.com.au does not respond to web requests. Using this domain, we will be unable to crawl your site or present accurate SERP information . What is wrong with this domain??
Technical SEO | | damientown0 -
is pointing to the same page that it is already on, is this a problem?
So we have a wordpress site with the all-in-one-seo-pack installed. I have just noticed in our crawl diagnostics that a canonical tag has been put in place on every single one of our pages, but they are all pointing to the pages that they are already on. Is this a problem? Should I be worried about this and delve more deeply to figure out as to why this has happened and get it removed? Thanks
Technical SEO | | cttgroup0 -
Advice on strange URL problem
I'm considering doing some pro bono work for a local non-profit and upon initial review they have a number of serious issues but there is one in particular I'd like to check my thinking on. The developer who set up the site some years ago implemented a javascript redirect on their root domain so that it redirects to: http://domain.com/wordpress This is wrong for all kinds of reasons and I want to recommend eliminating this redirect and getting rid of the 'wordpress' part of the path altogether. However, the site is quite established with good PR and they would take a hit by changing the path. I'd do 301 redirects to the new URLs that would not have 'wordpress' in the path in addition to other remediation. My question - is my thinking here good? It's worth it, right? The other option is just get rid of the weird redirect and keep 'wordpress' in the path but this seems unacceptable to me. Any opinions?
Technical SEO | | friendlymachine0 -
Confused about robots.txt
There is a lot of conflicting and/or unclear information about robots.txt out there. Somehow, I can't make out what's the best way to use robots even after visiting the official robots website. For example I have the following format for my robots. User-agent: * Disallow: javascript.js Disallow: /images/ Disallow: /embedconfig Disallow: /playerconfig Disallow: /spotlightmedia Disallow: /EventVideos Disallow: /playEpisode Allow: / Sitemap: http://www.example.tv/sitemapindex.xml Sitemap: http://www.example.tv/sitemapindex-videos.xml Sitemap: http://www.example.tv/news-sitemap.xml Is this correct and/or recommended? If so, then how come I see a list of over 200 or so links blocked by robots when Im checking out Google Webmaster Tools! Help someone, anyone! Can't seem to understand this robotic business! Regards,
Technical SEO | | Netpace0 -
Problem of printer friendly version.
For one of our client's side, most of the backlinks are going to printer friendly version page. I recommeded to him to use the canonical tag on printer friendly version pointing to other page. Luckily, while searching i came across this posts at - http://www.seomoz.org/q/solving-printer-friendly-version The solution recommended was this - <link type="text/css" rel="stylesheet" media="print" href="our-print-version.css"> My questions are - 1. what should i write in place of our-print-version.css Should it be print.css ? 2. Where do i place this code ? in which file ?
Technical SEO | | seoug_20050