Google cant read my robots.txt from past 10 days
-
http://awesomescreenshot.com/08d1s6aybc
hi, my robots.txt is
http://wallpaperzoo.com/robots.txt
google says it cant read and has postponed the crawl.. its been 10 days and no crawl.. please help me in solving this issue.. this is save with
-
Hi sir Google started fetching again
thank you for your help
-
What you can do is from within Google Webmaster Tools is to ask Google to fetch the URL of the sitemap and then 'submit it to the index'. That's one technique I've seen working before.
Google Webmaster Tools or Googlebot doesn't seem to crawl robots.txt files very often. Have seen it a couple of times where it could take days for them to crawl the file again after I've changed something.
-
hi sir, there was a problem on 23rd september in robots.txt which i fixed on 25th after then google bot dint come.. how much time it will take to come back again?
-
There's nothing wrong with your robots.txt file as far as I can see. It is likely that Google tried accessing the robots.txt file when your server was slow/down for a split second and that's why you are seeing the error. Are you sure that the site hasn't been crawled in 10 days? Have you checked your server log files to verify that? Regardless, once Googlebot does eventually come back, it should read the robots.txt file correctly and then you should be good to go.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
The Google Algo update that happened 1-8 is KILLING my rankings
Does anyone know what happened?? I have a great website, we ranked very highly for a slew of industry keywords, #1 in most of our top-money kws....and our keywords have been in freefall since the update. Help?!
Algorithm Updates | | Sean_Gutermuth0 -
Google SERPs showing blog comments in Answer Box?
I was recently researching Schema markup for local businesses and I was presented with an Answer Box that used blog comments as answers (at least I feel that's what they were attempting to show). This is what is says currently when I search for "schema markup hours" (screenshot also attached): 12 thoughts on “How to Use Schema Markup for Local SEO” Lauren says: March 11, 2013 at 2:22 pm. ... souleye says: March 11, 2013 at 3:29 pm. ... Daniel Bennett says: March 11, 2013 at 8:51 pm. ... sammy. says: ... Nathan says: March 11, 2013 at 11:53 pm. ... Rishav says: March 12, 2013 at 5:51 am. ... Paul Sherland says: ... keyword removed says: Right now it shows the time and date of the comment, but is this something that's new or has it been around? Thanks in advance! tp5y1od.png
Algorithm Updates | | TomBinga11250 -
Google is forcing a 301 by truncating our URLs
Just recently we noticed that google has indexed truncated urls for many of our pages that get 301'd to the correct page. For example, we have:
Algorithm Updates | | mmac
http://www.eventective.com/USA/Massachusetts/Bedford/107/Doubletree-Hotel-Boston-Bedford-Glen.html as the url linked everywhere and that's the only version of that page that we use. Google somehow figured out that it would still go to the right place via 301 if they removed the html filename from the end, so they indexed just: http://www.eventective.com/USA/Massachusetts/Bedford/107/ The 301 is not new. It used to 404, but (probably 5 years ago) we saw a few links come in with the html file missing on similar urls so we decided to 301 them instead thinking it would be helpful. We've preferred the longer version because it has the name in it and users that pay attention to the url can feel more confident they are going to the right place. We've always used the full (longer) url and google used to index them all that way, but just recently we noticed about 1/2 of our urls have been converted to the shorter version in the SERPs. These shortened urls take the user to the right page via 301, so it isn't a case of the user landing in the wrong place, but over 100,000 301s may not be so good. You can look at: site:www.eventective.com/usa/massachusetts/bedford/ and you'll noticed all of the urls to businesses at the top of the listings go to the truncated version, but toward the bottom they have the full url. Can you explain to me why google would index a page that is 301'd to the right page and has been for years? I have a lot of thoughts on why they would do this and even more ideas on how we could build our urls better, but I'd really like to hear from some people that aren't quite as close to it as I am. One small detail that shouldn't affect this, but I'll mention it anyway, is that we have a mobile site with the same url pattern. http://m.eventective.com/USA/Massachusetts/Bedford/107/Doubletree-Hotel-Boston-Bedford-Glen.html We did not have the proper 301 in place on the m. site until the end of last week. I'm pretty sure it will be asked, so I'll also mention we have the rel=alternate/canonical set up between the www and m sites. I'm also interested in any thoughts on how this may affect rankings since we seem to have been hit by something toward the end of last week. Don't hesitate to mention anything else you see that may have triggered whatever may have hit us. Thank you,
Michael0 -
Domain Name search in google not appearing
My hcg domain doesn't show up in google search. Shows up in new sand image search. If I wrap the domain name in quotes it shows up
Algorithm Updates | | noork0 -
You folks see the Google Analytics Real Time View (Beta)
I was pretty impressed by the amount of data they can present to you and how quickly the site gets it to you. Hit your GA account click the Home tab then click the Real Time (Beta) link in the navigation. It is pretty impressive. GoogleAnalytics.jpg
Algorithm Updates | | donford0 -
Google and Content at Top of Page Change?
We always hear about how Google made this change or that change this month to their algorithm. Sometimes it's true and other times it's just a rumor. So this week I was speaking with someone in the SEO field who said that this week a change occurred at Google and is going to become more prevalent where content placed at the "top of the fold" on merchant sites with products are going to get better placement, rather than if you have your products at top with some content beneath them at the bottom of the page. Any comments on this?
Algorithm Updates | | applesofgold0 -
How do I get the expanded results in a Google search?
I notice for certain site (ex: mint.com) that when I search, the top result has a very detailed view with options to click to different subsections of the site. However for my site, even though we're consistently the top result for our branded terms, the result is still only a single line item. How do I adjust this?
Algorithm Updates | | syount1 -
Strange Refferral URL coming in from Google
Hi, I've been monitoring my referral URL's coming in and today noticed they had changed. Previously when I clicked one it would be the google search result page - however now they all seem to be like this: http://www.google.co.uk/url?sa=t&source=web&cd=7&sqi=2&ved=0CHEQFjAG&url=http://www.mysite.com&rct=j&q=my%20keyword&ei=Bvc3TrbgB5G0hAfvqoSvAg&usg=AFQjCNFONDCPJDl3d2PYceYvale_cL7s4Q All these URL's immediately redirect to my website pages. Do you know what they are - they seem to be tracking URL's of some sort I am thinking?? Are they trying to analyse my site with respect to certain keywords?? Thanks
Algorithm Updates | | James770