Google cant read my robots.txt from past 10 days
-
http://awesomescreenshot.com/08d1s6aybc
hi, my robots.txt is
http://wallpaperzoo.com/robots.txt
google says it cant read and has postponed the crawl.. its been 10 days and no crawl.. please help me in solving this issue.. this is save with
-
Hi sir Google started fetching again
thank you for your help
-
What you can do is from within Google Webmaster Tools is to ask Google to fetch the URL of the sitemap and then 'submit it to the index'. That's one technique I've seen working before.
Google Webmaster Tools or Googlebot doesn't seem to crawl robots.txt files very often. Have seen it a couple of times where it could take days for them to crawl the file again after I've changed something.
-
hi sir, there was a problem on 23rd september in robots.txt which i fixed on 25th after then google bot dint come.. how much time it will take to come back again?
-
There's nothing wrong with your robots.txt file as far as I can see. It is likely that Google tried accessing the robots.txt file when your server was slow/down for a split second and that's why you are seeing the error. Are you sure that the site hasn't been crawled in 10 days? Have you checked your server log files to verify that? Regardless, once Googlebot does eventually come back, it should read the robots.txt file correctly and then you should be good to go.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Google ignores page title suffix?
Hi all, It's a common practice giving the "brand name" or "brand name & primary keyword" as suffix on EVERY page title. Well then it's just we are giving "primary keyword" across all pages and we expect "homepage" to rank better for that "primary keyword". Still Google ranks the pages accordingly? How Google handles it? The default suffix with primary keyword across all pages will be ignored or devalued by Google for ranking certain pages? Or by the ranking of website improves for "primary keyword" just because it has been added to all page titles?
Algorithm Updates | | vtmoz0 -
Is there a we to get Google to index our site quicker?
I have updated some pages on a website, is there a way to get Google to index the page quicker?
Algorithm Updates | | webguru20140 -
Am I the only one experiencing this Google SERP problem?
I perform Google searches every single day, sometimes several times in a day. These searches have nothing to do with being a marketer--they're simply as a consumer, researcher, person who needs a question answered, or in other words: a typical person. For about the past month or so, I have been unsuccessful at finding what I'm looking for on the first try EVERY SINGLE TIME. Yes, I mean it--every single time. I'm left either going all the way to the third page, clicking dozens of results and retuning to the SERPs, or having to start over with a differently worded query. This is far too often to be a coincidence. Has this been happening to anymore else? I know there was a recent significant algorithm update, right? I always look at algorithm updates through the eyes of an SEO, but I'm currently looking at it through the eyes for an average searcher, and I'm frustrated! It's been like trying to find something on Bing!
Algorithm Updates | | UnderRugSwept0 -
How can I check Googles Page Cache ?
Hi I use to have a handy tool in Firefox (Google Toolbar) that was very handy for checking page ranks and what date a page had been cached. For a while with the newer versions of Firefox I cannot seem to locate this useful tool, Can anybody recommend any useful tools for checking the above. Thanks Adam
Algorithm Updates | | AMG1000 -
Organic listing & map listing on 1st page of Google
Hi, Back then, a company could get multiple listings in SERP, one in Google Maps area and a homepage or internal pages from organic search results. But lately, I've noticed that Google are now putting together the maps & organic listings. This observation has been confirmed by a couple of SEO people and I thought it made sense, but one day I stumble with this KWP "bmw dealership phoenix" and saw that www.bmwnorthscottsdale.com has separate listing for google places and organic results. Any idea how this company did this? Please see the attached image
Algorithm Updates | | ao5000000 -
Is Google Rotating Good Matches?
I have a theory that Google may be trying to be fair to white-hat-seo sites that are doing the right things with blogging, linking, social media, etc. [ie that deserve equal good positioning] are being cycled to and from the first page, perhaps in a weekly or monthly basis. My theory would be that they are purposefully doing it to give those sites more equal exposure. My case: I've had top rankings for http://thedogbitelawyer.com for almost all of the important terms for dog bite lawyers for a couple of years now. When Penguin came out we lost some ground across the board, and identified that perhaps there was too much duplicate content left over from when I inherited the site. I reworked the site wording and link structure a bit and gained back positioning. Since that time we are up and down like a yo-yo on the top terms! Anybody else have this suspicion? If it's true, I don't need to stress, if we are bouncing around for other reason's I'd better keep stressing!
Algorithm Updates | | JCDenver0 -
How often do people use Google Product Search
I was was reading Tom Critchlow's excellent blog on how to rank well for Google Product Search. I'm trying to find out if there are stats on how often people use this feature in Google (since it is not listed on Google's main navigation). I'm working with a customer who has b-2-b products and am trying to determine the value of adjusting his ecommerce pages to appear on Google Product Search.
Algorithm Updates | | EricVallee340 -
Geolocation: Google only crawls from the US
A question was previously asked about geo-location and specifically if Google crawled from other countries. I could not locate the original question but wanted to share the below information. As of earlier this year Google only crawls from US IP addresses: http://www.youtube.com/watch?v=7paVYBgH0Hw
Algorithm Updates | | RyanKent1