Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can anyone explain this Google+ subdomain?
https://google36059.reachlocal.net/collections/featured This seems really odd to me. It appears to be a fully working Google+ page and I can access the links that are other Google+ pages, but it is a subdomain of Reach Local. Any ideas?
Link Building | | LocalEnthusiast0 -
Heavy Weight Backlink not Showing in Google Webmaster Tool
Hi Mozzers I got a nice backlink from Huffingtonpost already a couple of months ago from this page to coinstackr.com: http://www.huffingtonpost.com/2014/02/25/mt-gox-bitcoin_n_4854095.html However, the link doesn't show in the Google Webmaster Tools. Is that common? Cheers, Chris
Link Building | | Diderino0 -
What does it mean by followed links?
I don't understand what SEOmoz report means when it shows I have no followed links OR linking root domains. What does this mean and how can I fix it?
Link Building | | CapitolShine0 -
Does registering with the paid Yahoo directory have any effect on your google rankings?
I am trying to get quality back-links and I know Yahoo is one adn as it is a directory is still ok post penguin but can anyone tell me whether it is worth it or does Google ignore the Yahoo links? Thanks
Link Building | | ebowdublin0 -
How long does it take for google see my new backlins and take an effect on my rank?
how long does it take for google see my new backlins and take an effect on my rank?
Link Building | | ctam0 -
Link Building Management Tool That Syncs Sweetly With Google Analytics
Hi, I would like a link building management tool that syncs with Google Analytics, so that I can Easily see traffic referrals from link acquired sites Easily see traffic from keywords that were anchored in the link acquisition Browny points for the tool to be a CRM as well. Raven Tools cant do this. Does anyone know of a tool? Or do I keep building custom reports in Google Analytics! Thanks.
Link Building | | DigitalLeaf0 -
Do-Follow link from Linkedin/Facebook/Twitter/About/Google Profile, how to:
Tell us, a technician method of building do-follow link on high quality social websites. Thanks
Link Building | | leadsprofi0 -
Adding to intl versions of Google?
I ran the SEOmoz Competitive Link Finder tool for my URL and 5 of my competitors. The #1 result (where 3 of my competitors are listed but I'm not) is google.com.sg. Here's the URL where they are listed: http://www.google.com.sg/alpha/Top/Business/Investing/Derivatives/Options/Research_and_Analysis/ All 5 of us are US-based companies. Why would 3 of them be listed in the Singapore version of Google but not me? Is it common practice to add your URL to all local versions of Google, or should submission to the main google.com be enough?
Link Building | | scanlin0