Do I have a robots.txt problem?
-
I have the little yellow exclamation point under my robots.txt fetch as you can see here- http://imgur.com/wuWdtvO
This version shows no errors or warnings- http://imgur.com/uqbmbug
Under the tester I can currently see the latest version. This site hasn't changed URLs recently, and we haven't made any changes to the robots.txt file for two years. This problem just started in the last month. Should I worry?
-
Today it has a green check mark, and absolutely no changes were made to the website since I asked this question.
-
It could be that your server had a hard time when Google tried to view your robots.txt file that's why it wouldn't be able to fetch it. As long as this issue doesn't prevent Google anymore in the future it's not much to worry about.
-
That would make me feel more confident of a false error being reported. Time to closely monitor the crawl logs, look at server stats, and keep an eye on GWT for a change in the reporting/indexing. I would also go into the GWT forums and post, see if anyone is reporting a similar error these past couple days.
-
I can't post the domain but I know it is accessible.
When I go to the tester it shows the live robots.txt with no problems. I also can look at the server logs and see that it is being crawled, but being crawled less then Bing Crawls. Also the Bing Webmaster Tools is showing no problems.
-
Can you post your domain? Manually checking the robots.txt file would help.
I've checked many of my GWT accounts and I am not showing a sudden robots.txt error. It could be a false error, but I would take anything with the robots.txt file seriously. You'll want to make sure that it is in fact accessible to all the crawlers desired.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should a login page for a payroll / timekeeping comp[any be no follow for robots.txt?
I am managing a Timekeeping/Payroll company. My question is about the customer login page. Would this typically be nofollow for robots?
Technical SEO | | donsilvernail0 -
Exclude root url in robots.txt ?
Hi, I have the following setup: www.example.com/nl
Technical SEO | | mikehenze
www.example.com/de
www.example.com/uk
etc
www.example.com is 301'ed to www.example.com/nl But now www.example.com is ranking instead of www.example.com/nl
Should is block www.example.com in robots.txt so only the subfolders are being ranked?
Or will i lose my ranking by doing this.0 -
Robots.txt Download vs Cache
We made an update to the Robots.txt file this morning after the initial download of the robots.txt file. I then submitted the page through Fetch as Google bot to get the changes in asap. The cache time stamp on the page now shows Sep 27, 2013 15:35:28 GMT. I believe that would put the cache time stamp at about 6 hours ago. However the Blocked URLs tab in Google WMT shows the robots.txt last downloaded at 14 hours ago - and therefore it's showing the old file. This leads me to believe for the Robots.txt the cache date and the download time are independent. Is there anyway to get Google to recognize the new file other than waiting this out??
Technical SEO | | Rich_A0 -
Meta description tag problems according to an seo tool
hi, my site is www.in2town.co.uk I am using an seo tool to check on my site and how to improve the seo. The tool is here. http://www.juxseo.com/report/view/51ebf9deab900 for some reason it has brought up errors, it claims i have not got a meta description even though i have and have doubled checked in my source code the errors it has brought up is as follows, and i would like to know if this is a fault of the seo tool or am i doing something wrong Does the description tag exist?0/1 <a id="sub_toggle_12" class="sub_toggle contract_sub"></a>Hide Info Description Tag: Explanation: The meta description tag does not help your rankings but it is your opportunity to encourage prospects to click. The meta description should describe the content of your web page, include a strong call to action, and include your keyword. Action: Make sure you are using the meta description tag. It is found in the section of your page. checkboxIs there only one description tag?0/2<a id="sub_toggle_13" class="sub_toggle expand_sub"></a>More InfocheckboxIs your description less than 156 characters?0/1<a id="sub_toggle_14" class="sub_toggle expand_sub"></a>More InfocheckboxIs your keyword in the description tag?0/3 <a id="sub_toggle_15" class="sub_toggle expand_sub"></a>More Info it also says about the canocial tag which it claims i have more than one Is the canonical tag optmized? Is there only one canonical tag?0/4 <a id="sub_toggle_10" class="sub_toggle contract_sub"></a>Hide Info Explanation: You only need one of these to direct a search engine. Don't muddy the waters. Action: Make sure you only have one canonical tag. This only applies if you use the canonical tag. any help and advice would be greatregards
Technical SEO | | ClaireH-1848860 -
"Extremely high number of URLs" warning for robots.txt blocked pages
I have a section of my site that is exclusively for tracking redirects for paid ads. All URLs under this path do a 302 redirect through our ad tracking system: http://www.mysite.com/trackingredirect/blue-widgets?ad_id=1234567 --302--> http://www.mysite.com/blue-widgets This path of the site is blocked by our robots.txt, and none of the pages show up for a site: search. User-agent: * Disallow: /trackingredirect However, I keep receiving messages in Google Webmaster Tools about an "extremely high number of URLs", and the URLs listed are in my redirect directory, which is ostensibly not indexed. If not by robots.txt, how can I keep Googlebot from wasting crawl time on these millions of /trackingredirect/ links?
Technical SEO | | EhrenReilly0 -
Auto genrated content problem?
Hi all, I operate a Dutch website (sneeuwsporter.nl), the website is a a database of European ski resorts and accommodations (hotels, chalets etc). We launched about a month ago with a database of about 1700+ accommodations. Of every accommodation we collected general information like what village it is in, how far it is from the city centre and how many stars it has. This information is shown in a list on the right of each page (e.g. http://www.sneeuwsporter.nl/oostenrijk/zillertal-3000/mayrhofen/appartementen-meckyheim/). In addition a text of this accomodation is auto generated based on some of the properties that are also in the list (like distance, stars etc). Below the paragraph about the accommodation is a paragraph about the village the accommodation is located in, this is a general text that is the same with all the accommodations in this village. Below that is a general text about the resort area, this text is also identical on all the accommodation pages in the area. So a lot of these texts about the village and area are used many times on different pages. Things went well at first and every day we got more Google traffic, and more and more pages. But a few days ago our organic traffic took a near 100% dive, we are hardly listed anymore and if we are at very low places. We expect the Google gave us a penalty. We expect this to be the case because of 2 reasons: we have auto generated text that only vary slightly per page we re-use the content about villages and area's on many pages We quickly removed the content of the villages and resort area's because we are pretty sure that this is definitely something Google does not want. We are less sure about the auto generated content, is this something we should remove as well? These are normal readable text, they just happen to be structured more or less the same way on every page. Finally, when we made these and maybe some other fixes, what is the best and quickest ways to let Google see us again and show them we improved? Thanks in advance!
Technical SEO | | sneeuwsporter0 -
Problem generating backlinks
Hello everyone, Over the past couple of days I have been using a variety of different free and paid programs to check how many backlinks I have. I have used at least 15 different ones, SEOmoz is the newest and am awaiting those results. All but one of those programs have said that I have 0 backlinks. The one said I have 11. The thing is...over the past month I have submitted my site to over 500 PR0 directories, 250+ PR3-PR7 directories, have made well over 75 article submissions to over 20 websites(almost all of them were approved and up and running, and some of these websites have a PR of 7), a couple dozen Press Releases, set up profiles linking to my site on over 50 forums, set up profiles linking to my site on a variety of different Web 2.0 sites, and still nothing. I have been doing this every day for over a month. Anyone have any ide?
Technical SEO | | tarik30010 -
My home page 301 redirects - is this an SEO problem
When ever a browser calls my site canineconcepts.co.uk, it is automatically 301 redirected to canineconcepts.co.uk/en I am not sure if I should be concerned about this from an SEO perspective or not. Any thoughts?
Technical SEO | | CanineConcepts0