Google cant read my robots.txt from past 10 days
-
http://awesomescreenshot.com/08d1s6aybc
hi, my robots.txt is
http://wallpaperzoo.com/robots.txt
google says it cant read and has postponed the crawl.. its been 10 days and no crawl.. please help me in solving this issue.. this is save with
-
Hi sir Google started fetching again
thank you for your help
-
What you can do is from within Google Webmaster Tools is to ask Google to fetch the URL of the sitemap and then 'submit it to the index'. That's one technique I've seen working before.
Google Webmaster Tools or Googlebot doesn't seem to crawl robots.txt files very often. Have seen it a couple of times where it could take days for them to crawl the file again after I've changed something.
-
hi sir, there was a problem on 23rd september in robots.txt which i fixed on 25th after then google bot dint come.. how much time it will take to come back again?
-
There's nothing wrong with your robots.txt file as far as I can see. It is likely that Google tried accessing the robots.txt file when your server was slow/down for a split second and that's why you are seeing the error. Are you sure that the site hasn't been crawled in 10 days? Have you checked your server log files to verify that? Regardless, once Googlebot does eventually come back, it should read the robots.txt file correctly and then you should be good to go.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Post-August Google Updates High volume keyphrases rankings mess
Howdy, fellow mozzers, Couple of our clients keep losing rankings, slowly but surely, which seem to correlate to August Google updates. No major changes to the websites have been done - neither code, backlink profile nor content. It seem to affect higher search volume keyphrases, while other longer tail keyphrases are climbing up. The screenshots of the visibility rankings are attached below. Any ideas/news/thought about the Google update itself or what might be happening? I know that the Medic update was overall quality, yet it seems to be that these websites in particular are going down after Birthday update, not Medic itself. Google and da mighty Internets being quite.. Help! Links to screenshots (includes higher volume keyphrases only): https://dmitrii-regexseo.tinytake.com/sf/MzAyNjg0MV85MDczNDkz https://dmitrii-regexseo.tinytake.com/sf/MzAyNjgzN185MDczNDY1
Algorithm Updates | | DmitriiK0 -
When Google is going to penalise webpages with autoplaying videos?
Hi, One of the most disgusting experiences while browsing web pages is the auto playing anonymous and highly non relevant ad videos at some corner of the page which buffer mostly and automatically plays. There are familiar websites which been following this activity. I don't think Google has ever rolled out an update to penalise these. Any idea why? Thanks
Algorithm Updates | | vtmoz0 -
Many meta descriptions ignored by Google
Hi all, We have recently added the meta descriptions for more than 50 pages of our website. It's been more than a week and all the pages have been indexed. But still I can see most of the pages in Google results didn't show up with recently added meta description, but the content from page like how it used to be. I wonder what's wrong with this scenario. Please guide of someone aware of this. Thanks
Algorithm Updates | | vtmoz0 -
Does Google’s Algorithm Populate Answer Boxes with Its Own Independent Research?
If you search 'best games to play for youtube' you get an answer box with answers pulled independently from the article at hand. Here's an image: http://imgur.com/a/S0j9B Here are all the games from my article, in the order in which they appear. Google's chosen games for Answer Box are bolded: Battlefield 1 Bloodborne GTA V FiFA 16 TrackMania Turbo Garry’s Mod League of Legends Call of Duty: Black Ops III Tom Clancy’s The Division Overwatch Just Cause 3 Counter-Strike: Global Offensive Brawlhalla Rocket League Dark Souls III Unravel Firewatch GoldenEye 007 (this was put in as a joke, but coded as an H2 nonetheless) Destiny Dead by Daylight Fallout 4 Undertale No Man’s Sky Minecraft As you can see, Google is choosing which games to display to its searchers. My Crazy Egg data shows that these were not picked by click volume (each of these H2s are hyperlinked), which means Google must be using some other popularity metric, such as its own search volume data or external sales data. I wrote this up in a post on my site, for anybody who's curious.
Algorithm Updates | | Edward_Sturm1 -
How to check which site performing well in google organic?
Hi All, Is it possible to check sites via any tool which sites performing good in google organic? Any site ... Is it possible via Alexa? My Concern is majorly for UK Ecommerce site... Thanks!
Algorithm Updates | | pragnesh96390 -
How to keep damage low on Google after the change of URL's
Hi Peeps, Hope someone can shed a light on this and show a guidance if possible. We are going to move our sites to shopify and shopify's URL's cannot be customized to match exactly like our current URLs. What steps do I need to take so google knows the URL's are changed. Domain will be the same. Thank you in advanced.
Algorithm Updates | | cemalcebi0 -
Google dance/over optimized/paranoid?
Hi guys, hope your all OK and thanks in advance for taking a nosey at this. OK where to start - my rankings for the last 12 months have progressively improved every week, usually of the 300 KWs i track the last few months has seen approx 70 up/70down per week, but the improvements usually outweigh the declines. This week I saw a sudden drop though - 35 improvements and 112 declines. The strange thing was though, the improvements came on the more competitive KWs, and the less competitive words I haven't done much or any back linking for dropped. Seems silly me asking this question when I run that through my head ofcouse KWs you don;t work on will drop like flies? It should be plainly obvious those words would drop off but all have been improving on there own slowly over the last 6/7 months. Now if this was a penalty (nothing showing in webmaster tools) I would have expected it to come through on my KWs I have over done the backlinking for, but these are the 1's that improved. So is it just the Google Dance? I normally see some words such as the big 1 we target DJ Equipment go from position 13 - 24 can change hourly sometimes! Could it just be quite a few have dropped all at once and will pop back up this week? Also if anyone could give us any pointers in general on where you think we should be taking our SEO it would be much appreciated. I know we have been a little lazy with our backlinking and could do with some much better/ industry related websites linking to us, and there are title tags/metas on product page that need sorting.. aside these couple of issue's? DJs Only
Algorithm Updates | | allan-chris0 -
Google Update on the 6th July
Hi Mozzers, Has anyone noticed a Google update on the 6th July? A price comparison site I optimise has fallen off the SERPs for most generic terms, however still getting traffic for longer tail phrases. Cheers Aran
Algorithm Updates | | Entrusteddev0