Google cant read my robots.txt from past 10 days
-
http://awesomescreenshot.com/08d1s6aybc
hi, my robots.txt is
http://wallpaperzoo.com/robots.txt
google says it cant read and has postponed the crawl.. its been 10 days and no crawl.. please help me in solving this issue.. this is save with
-
Hi sir Google started fetching again
thank you for your help -
What you can do is from within Google Webmaster Tools is to ask Google to fetch the URL of the sitemap and then 'submit it to the index'. That's one technique I've seen working before.
Google Webmaster Tools or Googlebot doesn't seem to crawl robots.txt files very often. Have seen it a couple of times where it could take days for them to crawl the file again after I've changed something.
-
hi sir, there was a problem on 23rd september in robots.txt which i fixed on 25th after then google bot dint come.. how much time it will take to come back again?
-
There's nothing wrong with your robots.txt file as far as I can see. It is likely that Google tried accessing the robots.txt file when your server was slow/down for a split second and that's why you are seeing the error. Are you sure that the site hasn't been crawled in 10 days? Have you checked your server log files to verify that? Regardless, once Googlebot does eventually come back, it should read the robots.txt file correctly and then you should be good to go.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Any recent updates from Google or community on sub domains vs sub directories?
Hi all, This has been a debate for years and I have noticed most of the SEOs suggest to go or switch to sub directories instead of sub domains. Still is this the same or any new updates from Google or SEO community? We have moved a sub domain to sub directory last year. The result was sub directory content started ranking good; but no change in website rankings. Because of moving sub domains to sub directories, will the linkjuice/PR of the website gets diluted as the number of pages increases which will takeaway soe authority? Thanks
Algorithm Updates | | vtmoz0 -
Is there any way to prevent Google from using structured data on specific pages?
I've noticed that Google is now serving what looks like host-specific video cards on mobile for our site. Is there any way to control which videos are included in these lists without removing the structured data on those clip pages or user pages? We don't want to noindex those pages but we don't want content from those pages to appear as video cards. 1kzPW
Algorithm Updates | | Garrett570 -
How to keep damage low on Google after the change of URL's
Hi Peeps, Hope someone can shed a light on this and show a guidance if possible. We are going to move our sites to shopify and shopify's URL's cannot be customized to match exactly like our current URLs. What steps do I need to take so google knows the URL's are changed. Domain will be the same. Thank you in advanced.
Algorithm Updates | | cemalcebi0 -
Which Google Snippet to use? Products or Review
A "product" page that includes "user reviews/ratings" and "comparison prices", which Google snippet do/should you use? Thanks in advance
Algorithm Updates | | righty0 -
Can Google display a diffrent page title?
Hi if I search google UK for the phrase car leasing, google returns my listing as Car Lease Deals However the same search on Yahoo or Bing bring back Contract Hire | Vehicle & Car Leasing Deals | Car Lease Deals this is the real page title. Why would this happen? Thanks Andy
Algorithm Updates | | First-VehicleLeasing0 -
Kilobytes downloaded per day: poss causes for MAJOR drop?
mid august, i went from averaging about 7 million kb/day but since have been at about 800 thousand. time spent downloading each page went up maybe 30% but pages crawled stayed the same. apart from the speed issue, don't think we've done anything to trigger this (none of the usual suspects like link structure, errors, sitemap, or new code). any other ideas? and is this disastrous for my indexing or am i over-reacting. thanks for your help
Algorithm Updates | | babble0 -
Was I Kicked Off Google Page One by Panda/Farmer?
Took over this site in March. Got a Panicked call from client Mid-March that all of a sudden keywords that put the site on Page One weren't working. There are still 9 that work, but apparently there were more. A large percentage of the backlinks are from Article Directories and Link Farms. Is this my problem? Also, a large percentage of the 149 pages suffer from keyword stuffing and were obviously written for Search Engines and not people. How much of a difference does that make?
Algorithm Updates | | reeljerc0 -
Do we have a timeline of google, bing updates
I thought it would be handy if we had a timeline with dates of any updates to the algo's.
Algorithm Updates | | AlanMosley
Does one exists here at SEOMoz or elsewhere.
Thanks3