Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hi. One of our competitors is ranking ahead of us on Google. Our site has a much stronger authority and much more quality links than this competitor. Would anyone have any explanations for this? Thanks
Hi Guys - I've noticed recently that one of our competitors is ranking extremely well on Google, while we seem to be fluctuating, along with other competitors, for our main keywords. Our site has a much stronger authority and much more quality links than this competitor and traffic is continuously increasing each month. Would anyone have any explanations as to why we may be seeing this? Any help would be greatly appreciated, thanks.
Link Building | | barryhq0 -
Followed links from one of our biggest channel?
Hi everyone, I have a new client that's a huge company and has a disastrous ranking. It's a Human Resources company that among other things, they post job offers on their site in which people have to register to apply. As a channel to outreach more people, they are using a popular site for job offers on the country. There, the job position is described and they tell users to apply within this channel or on their own site. But there's no actual link to their site. Will Google find it spammy to post their site's link for that specific job offer on this channel on every job description? No competitor is doing it so it made me doubt a little bit. NOTE: The job offer channel doesn't allow us to "nofollow" a link. All the links are followed. Thanks to any suggestion.
Link Building | | guillermoga0 -
Is something strange going on with Google last few days?? Seeing strange results for competitive keywords
I am seeing some very strange results for competitive keywords. If I search on google.co.uk for "website design" which I feel is a pretty tough keyword. I see a site called http://www.techdesigns.co.uk/ If you go to the site its a laptop review site with no mention of website design on the site. They seem to be an Amazon Affiliate They have 23,597 links saying website design but mostly from adult websites. I have seen this on a few result related to SEO but dont want to mention any names in case we upset anyone on the forum. How can they rank for such a competitive keyword without onpage? Surely Panda should have hit this site? Why have they not been hit by Penguin so many adult links? Sorry if this is anyones site on here I am not looking to get you banned or hit just stumps me why this site ranks.
Link Building | | tempowebdesign0 -
Website not appearing anywhere on google
Hello, we launched www.comparetravel.ie in June but it is not showing up on google search, we have submitted it to google twice but still no sign of our site. I thought it would at least show up for the "Compare Travel" search term, it only shows up when you type the full website address into google. Any ideas on what we are doing wrong? Alll advice greatly appreciated. (The website is Irish based) Thanks P
Link Building | | CompareTravel0 -
Got Google Warning But No Penalty - Should I Ask for Reconsideration?
I got the following warning from Google yesterday: "Google Webmaster Tools notice of detected unnatural links to [...] We encourage you to make changes to your site so that it meets our quality guidelines. Once you've made these changes, please submit your site for reconsideration in Google's search results. " But my website didn't lose any traffic or ranking so far. Anyway, I'm currently cleaning up my link profile. My question is, since the website didn't seem to have a penalty, should I still submit the website for reconsideration once the clean up is done? If I clean up my link profile, would that warning just go away? Should I avoid a manual review of my website? Thanks for your help.
Link Building | | sbrault740 -
Google and low quality guest posts
In the aftermath of the Penguin and Panda monsters a lot of blog networks seem to have been devalued by Google. It seems logical that spammers will flock to low quality guest posting next. How do you think Google will determine which guest post links to devalue? Thanks!
Link Building | | ProjectLabs0 -
Product Giveaways and No-Follow
If a blogger is given a product from a company for a giveaway contest on their blog, does Google require the link to the company have a no-follow. This seems like a gray area but I can't find any guidelines put out by Google about this.
Link Building | | SparkplugDigital0