Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
slug Link redirect to subdomain?
Hi !
Link Building | | Leviiii
Im Levi new here and new in the world of SEO, please dont judge if my questions are silly. Back on the days when the site was built we thought it is a good ideea to have subdomains that together with the domain name represent our main keywords.
ex. https://stansted.tonorwich.uk, https://heathrow.tonorwich.uk, https://luton.tonorwich.uk, https://gatwick.tonorwich.uk. There is content on this subdomains, would it make any difference from SEO perspective if we create slugs that redirect to these subdomains? for example creating https://tonorwich.uk/taxi-minibus-vip-tesla-norwich-to-stansted that redirects to https://stansted.tonorwich.uk ? Or better create these slugs with slightly different content?
Any ideeas would be appreciated.
Thanks in advance!0 -
301 passing authority?
Does anyone have experience of 301's in connection with SEO or more specifically passing authority? Recently purchased a small business that had three small websites. My main concern is all three websites do rank for some fairly important key phrases, but there is very little direct overlap. I was hoping by using 301's I would be able to refocus the authority to get the main site ranking.
Link Building | | seoman100 -
Very old site wanting a revamp - how to handle all the redirects/URL changes
Hi all, I have a very very old site, that requires a facelift (going from an old clunky CMS to wordpress). A problem I know I am going to face is that this site has a very poor URL structure (www.site.com/section/1/section/ etc) which I plan on cleaning up. but how do I handle the 100+ backlinks this website has point to such links? If someone could point me in the direction of an article or such that would be great. Thanks,
Link Building | | Greenroads0 -
External Followed Links
Hello all, I was wondering if anyone could help me. I have been inputting links for a few months on various different sites but it doesn't seem to be generating links on Open Site Explorer. Does anyone know a reason for this? Thanks in advance, Beth
Link Building | | cmuknbb0 -
01 November 2013 - A possible Google's update??
Hello Guys! On 01 November 2013, one of our website's traffic has been dropped by more than 70%. We had added plenty of domains to the disavow tool, I am just wondering if its a Google's update or the disavow tool has destroyed the site? Any ideas? 9p3sN3p.jpg
Link Building | | TheSEOGuy10 -
What is the difference between external followed links and internal followed links.
I have almost 3,000 external links followed and only a tiny amount of internal followed links. My website is doing really badly on google at the mo. I am also an SEO beginner and learning new stuff every day, but am not sure exactly what the difference is! Thanks!
Link Building | | heleni0 -
Google sending warnings about Artificial or Unnatural links in Google Webmaster Central
Has anyone seen warnings about Artificial or Unnatural links notice show up in their Google Webmaster central yet? I just looked at each of our clients and had not but after reading the Search Engine Land article at http://searchengineland.com/google-warning-more-about-bad-link-networks-117079 I was wondering what your thoughts were regarding this topic? We are not purchasing links for our clients sites but definitely going out their and building links. Article concerned me that I may run into issues for my link building clients. Thoughts?
Link Building | | jfeitlinger0 -
Are backlinks to a domain that redirects useless?
I am working on trying to get backlinks. We are currently redirecting a few Domains to our current domain name. One of the Domains that is redirecting is the main Domain, we will be advertising in the near future. Can I send backlinks to the redirected URL or does it only carry link juice if it is to the main Domain?
Link Building | | hfranz0