Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I set "no follow"
I see some very spammy site linking to my site. How do I tell google to "no follow" those links in regards to my site?
Link Building | | Kuhliff0 -
Very old site wanting a revamp - how to handle all the redirects/URL changes
Hi all, I have a very very old site, that requires a facelift (going from an old clunky CMS to wordpress). A problem I know I am going to face is that this site has a very poor URL structure (www.site.com/section/1/section/ etc) which I plan on cleaning up. but how do I handle the 100+ backlinks this website has point to such links? If someone could point me in the direction of an article or such that would be great. Thanks,
Link Building | | Greenroads0 -
Is there a downside to "closing" a domain and 301'ing to to your main domain?
We have a domain which was optimized for a specific keyword. It does very well for that keyword, but seems like it has in turn caused our primary domain to be penalized (back to page 3) for that same keyword (We think they've figured out the connection). The optimized domain (only one page) has links on it to our primary domain, so I don't want to delete it as that's where our traffic currently comes from. But if I deleted it and 301'ed it to the primary domain is there a downside? Alternatively I could disavow it, but if the search engines know it's the same owner will they hold that against all domains? Or, alternatively again. Should I turn it into a more substantial website? Which could in turn remove it's "spamminess". Any suggestions are welcomed!
Link Building | | absoauto0 -
How to increase search engine visibility on google , any new ideas ??
Hello friends , I have a shopping site how can i increase google ranking in google serp. Now my page is in 700 . Thanks
Link Building | | BiswaM220 -
Backlinks Redirected Temporarily (New to Old)
I have a website of which we are being forced to change the domain name and soon after give the control of the domain name to the corporate office. The corporate office will eventually put a new website up on it. We would like to build more backlinks without the threat of losing all of these backlinks to the corporate site. We were considering to start building backlinks to the new domain with temporary redirects to the old domain but have recently been advised against doing this (threat of Google looking down on this technique). Should we continue to build back links to the old domain, or should we 302 redirect the links to the old domain? Any advice is greatly appreciated!
Link Building | | wishmedia0 -
Google Webmaster Link drop
Checking Google Webmaster it looks like my links dropped in half? Yet I have been ranking very well the past 2-3 days? Is there any explanation for this? I have been doing Pr's, Blogs, manual link building, and concentrating on the social side of it as well, facebook, twitter, google+ can that have anything to do with better rankings? or is it still considered a minor part? Thanks!
Link Building | | TP_Marketing0 -
Google Webmaster Tools Link Count
What are your thoughts on the link information provided by GWMT? It often takes quite a bit of time for the links to show up, and some quality links are never shown. Is it cause for concern if your link count is lower in GWMT than, say OSE? Any implications worth noting?
Link Building | | ericbramlett0 -
Should I 301 Redirect The index.html Page?
Hello, As an experiment I am trying to work with an SBI site but am noticing index.html is being forced on me as the homepage. As I understand it's good practice to 301 redirect the index.html page to the to the actual home page. Can someone please confirm that I am right in thinking this? Here is an example: http://www.anguilla-beaches.com/index.html
Link Building | | umtmedia
http://www.anguilla-beaches.com/ According to yahoo 773 links are pointing to the index.html page and 771 links are pointing to the / page. I'm sure Google knows that these are the same pages and does not consider them as duplicate but surely the index.html page should be redirected right? am I going mad here? Now if we look at this site: http://www.jugglenow.com the homepage has 198 links according to yahoo
http://www.jugglenow.com/index.html this page also has 198 links? now I am confused, are these 2 the same page? On this page: http://www.jugglenow.com/juggling-balls.html notice how there is no link to the actual homepage but there is a link to the index.html page. Am I missing something here because I can't figure it out, surely there should be just 1 homepage without a index.html page. Any help would be appreciated0