Do I have a robots.txt problem?
-
I have the little yellow exclamation point under my robots.txt fetch as you can see here- http://imgur.com/wuWdtvO
This version shows no errors or warnings- http://imgur.com/uqbmbug
Under the tester I can currently see the latest version. This site hasn't changed URLs recently, and we haven't made any changes to the robots.txt file for two years. This problem just started in the last month. Should I worry?
-
Today it has a green check mark, and absolutely no changes were made to the website since I asked this question.
-
It could be that your server had a hard time when Google tried to view your robots.txt file that's why it wouldn't be able to fetch it. As long as this issue doesn't prevent Google anymore in the future it's not much to worry about.
-
That would make me feel more confident of a false error being reported. Time to closely monitor the crawl logs, look at server stats, and keep an eye on GWT for a change in the reporting/indexing. I would also go into the GWT forums and post, see if anyone is reporting a similar error these past couple days.
-
I can't post the domain but I know it is accessible.
When I go to the tester it shows the live robots.txt with no problems. I also can look at the server logs and see that it is being crawled, but being crawled less then Bing Crawls. Also the Bing Webmaster Tools is showing no problems.
-
Can you post your domain? Manually checking the robots.txt file would help.
I've checked many of my GWT accounts and I am not showing a sudden robots.txt error. It could be a false error, but I would take anything with the robots.txt file seriously. You'll want to make sure that it is in fact accessible to all the crawlers desired.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Https problem on google result.
Hello everyone. My problem is SSL certificate... Send all links to google, after google shows https link no problem. But a few minutes ago my home page link not have an SSL..
Technical SEO | | dalapayal
Please check this page : https://www.bodrumtransfermarket.com Where do I make a mistake? Thanks for all...0 -
Robots.txt on subdomains
Hi guys! I keep reading conflicting information on this and it's left me a little unsure. Am I right in thinking that a website with a subdomain of shop.sitetitle.com will share the same robots.txt file as the root domain?
Technical SEO | | Whittie0 -
3,511 Pages Indexed and 3,331 Pages Blocked by Robots
Morning, So I checked our site's index status on WMT, and I'm being told that Google is indexing 3,511 pages and the robots are blocking 3,331. This seems slightly odd as we're only disallowing 24 pages on the robots.txt file. In light of this, I have the following queries: Do these figures mean that Google is indexing 3,511 pages and blocking 3,331 other pages? Or does it mean that it's blocking 3,331 pages of the 3,511 indexed? As there are only 24 URLs being disallowed on robots.text, why are 3,331 pages being blocked? Will these be variations of the URLs we've submitted? Currently, we don't have a sitemap. I know, I know, it's pretty unforgivable but the old one didn't really work and the developers are working on the new one. Once submitted, will this help? I think I know the answer to this, but is there any way to ascertain which pages are being blocked? Thanks in advance! Lewis
Technical SEO | | PeaSoupDigital0 -
Wordpress Hatom problem
Hi, in Webmaster Tools i receive the following warnings: hatom-feedhatom-entry:Warning: At least one field must be set for HatomEntry.Warning: Missing required field "entry-title".Warning: Missing required field "updated".Warning: Missing required hCard "author".I googled a few strategies how to solve this problem but is it for SEO purpose really necessary to edit Theme core code to satisfy google's warnings?
Technical SEO | | reisefm0 -
Do I have a canonical problem?
Does this site www.davidclick.com have a canonical problem because the home page can be requested via 3 different urls http://www.davidclick.com/
Technical SEO | | Nightwing
http://davidclick.com/
http://www.davidclick.com/index.htm but I'm confused in terms of applying a fix for example all advice here http://support.google.com/webmasters/bin/answer.py?hl=en&answer=139066#1 says i need to identify the duplicate files and add So my question is please if I do have a canonical problem how can i fix it when I only have one file for my home page, there are no duplicates 😞 Any insights welcome 🙂0 -
403 forbidden, are these a problem?
Hi I have just run a crawl test on screaming frog and it is showing quite a few 403 forbidden status codes. We are showing none of these in webmaster tools, is this an issue?
Technical SEO | | jtay1230 -
Multiple redirects a problem?
When product is sold out I will 301 redirect to a category page if a similar product is not available, but now our web developer has changed all the url's of the category pages so I need to redirect them all to the new category pages but that means there are some products that are first being redirected to the no longer existent category and then being redirected again to the new category page. This seems like it might me be a problem having two 301 redirects so I wanted to find out for sure if it is. Unfortunately our system for redirecting pages is archaic so it will be difficult and time consuming to go back and redo all the redirects that are going to pages that no longer exist so I wanted to get some additional opinions before I do that.
Technical SEO | | KentH0 -
Mobile site - allow robot traffic
Hi, If a user comes to our site from a mobile device, we redirect to our mobile site. That is www.mysite/mypage redirects to m.mysite/mypage. Right now we are blocking robots from crawling our m. site. Previously there were concerns the m. site could rank for normal browser searches. To make sure this isn't a problem we are planning on rel canonical our m. site pages and reference the www pages (mobile is just a different version of our www site). From my understanding having a mobile version of a page is a ranking factor for mobile searches so allowing robots is a good thing. Before doing so, I wanted to see if anyone had any other suggestions/feedback (looking for potential pitfalls, issues etc)
Technical SEO | | NicB10