Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What about a No-record backlink in the eye of Google
I have an uncertainty - when I make a backlink as a piece of SEO in some site when I reviewed similar couple of days after. It hasn't filed and I checked its robots record. It appearing Client specialist: Mediapartners-Google Disallow: User-Agent: * Disallow: However, is this make any backlink uphold or only this with the end goal of not ordering in google. I make it straightforward - "Is this sort of backlink creation my site Best Offset Smokers uphold my SEO action or Not?" In this No-record site. Flag Share
Link Building | | seotools1030 -
Fitness- china.com 301 Redirects
After I switched to 301, will the backlinks obtained by the previous domain name be transferred to the new URL? our Weight Machine new URL https://www.fitness-china.com/weight-machine
Link Building | | ahislop5740 -
Back link from site with DA of 72 to a website domain. Clicking on the link redirects to our website not the attended one.
Hi,
Link Building | | JIMBO16
I've ran a back link check and discovered a good back link to a site which then gets redirected back to my company's website. I have a feeling that an old SEO agency has purchased a small website which has a decent link back from a relevant organisation with a high Domain authority and then redirects the domain to our website to get the link juice. What are your thought on this? Is this really bad practise and possibly damaging? Thanks, Jim0 -
301 redirect lost page authority & total links how ensure it passes on.
hello, I recently changed from www to non www. I have redirected my whole website so if I type www. it goes to the non-www page. doing a open site explorer I noticed my www. still has the page authority 14/100 with 6 links. where as the non-www page has 1/100 page authority and no links. how do I get the www. to mirror the non-www links?
Link Building | | finnmoto
eg if you goto http://www.finnmoto.com.au/mens-motorcycle-kevlar-hoodie/ it will goto the non-www version automatically, so why does the non-www replicate the page authority / total links attached picture for reference.
Google webmaster has been selected for non-www, its been about 2 weeks.
my domain authority i think about about 21 or more now its 19 aswell. I have a feeling:
nofollow or rel=canonical has something to do with it but Ill need read into what that does and how to apply. any ideas? uRiJ2Ul.png?10 -
301 Redirect and Google...
Hi all Recently, Google has indexed the landing page url for our product differently. This has just happened. So now, when you used search for "DocRead" you would find the top result being : "http://www.collaboris.com/products/policy-and-procedure-management-software" which is what i want. However, now it's indexing it as "http://www.collaboris.com/products" this page is simply a redirect to our product page as it's part of the menu. If I look in fiddler when requesting http://www.collaboris.com/products I can see its doing a 301 redirect, which is fine. Why is Google storing the wrong url ? I also noticed we dropped in our rankings quite significantly last week.
Link Building | | MarkQJones0 -
How to make google index the page title has changed ?
Google has indexed all my websites, but I have to change some title in the website but when checking google does not update the change.
Link Building | | tuananh0688
Please help me .
My site : http://blog.addme.vn0 -
Date specific keywords, what to do for following year?!?
Hi, Lets say I have a page with the keyword 'film releases 2012', I rank the term and it picks up some back links and social signals. The year ends and search begins for 'film releases 2013'. Do I: 1. Create a new page targeting 'film releases 2013' and redirect the old page to the new one. 2. Just have one page with the url .co.uk/film-releases/ and when the year ends for 2012 just change the 2012 to 2013 (title, h1 tags etc). I'm 99% sure number 2 is the best option but the only negative is the url does not contain the year. I could just have the 1 page with the date in the url and when it enters the new year change the url to contain the new date but that would break any backlinks, so I could redirect anyone trying to get to the old url to the new one which should pass the PR across. But im guessing I would lose some PR in the process so maybe not having the date in the url would not effect SEO to much so rolling with number 2 is best bet. Cheers
Link Building | | activitysuper0 -
301 redirect and how linking domains are counted
I have a domain with several great links to it and I decided to change it to a new domain. I did a 301 redirect to the new domain. My question: Let's say youtube.com is linking to the old domain that is being redirected. Do I want to get a new link from youtube to the new domain as well, or is that pointless since there is already one to the old domain? Thanks,
Link Building | | DavidCurrier
Dave0