Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Back link from site with DA of 72 to a website domain. Clicking on the link redirects to our website not the attended one.
Hi,
Link Building | | JIMBO16
I've ran a back link check and discovered a good back link to a site which then gets redirected back to my company's website. I have a feeling that an old SEO agency has purchased a small website which has a decent link back from a relevant organisation with a high Domain authority and then redirects the domain to our website to get the link juice. What are your thought on this? Is this really bad practise and possibly damaging? Thanks, Jim0 -
Reporting a Link Scheme to Google
Hi Mozzers, Two questions...
Link Building | | FDAitsupport
There is a negative article showing up in the SERPS for my boss name which we've been doing some rep management to get rid of. Well, someone started building links to that property recently, and it has pushed the result to the #2 spot. I did some research, and the property has never had a single link pointed to it until recently. There are 7 referring domains, all of which use the same theme, and have the same author for each article. Each property has over 1,000 articles in not even 2 years. All articles are based around the same topic. The property they are promoting negatively went from 0 backlinks to 55 in under 2 months. My Question is, is it okay to report this to Google? It seems like a cut and dry example of a link scheme. Second question, I have recently seen an increase of spammy links showing up in ahrefs. These links were likely built by a predecessor of mine as far back as 2010. I had been letting them fall off naturally, but now they are coming back (around the same time the above mentioned link scheme began). Do you think someone is re-indexing my links as a sort of attack? Or is it possible Google is re-indexing them? I'm going to contact the webmasters where I can, but seriously considering using disavow tool. My rankings started dropping when the links started getting indexed. And continue to drop. Negative SEO, or Googles recent "Quality Update"? Thanks for any and all input. Ryan0 -
How can I tell if a site is trustworthy and is not / hasn't been penalized by Google?
Hello Moz Community, I'm looking to do some link building for a client and I would like to know if there's a way to find out if a website is trustworthy and is not or hasn't been penalised by Google. Thank you.
Link Building | | CosminC0 -
Google Authorship and Guest Posts
I'm including my google author details at the bottom of guest posts with a link to my site. Is this not giving google a clear signal the that it is a guest post by me and although the site owner has found it worthy enough to include the post on their site it's not really an 'editorially given link' in the true sense of the term?
Link Building | | SamCUK0 -
Difference between rel="nofollow" and rel="no follow"
Does rel****=****"nofollow" mean the same thing as rel=****"no follow"? I mean written in one word or two words. Can Google feel the difference? Thanks
Link Building | | DDL0 -
Same Page, Multiple URLS- Canonical tag? Redirect?
I just read this article about how google views multiple versions of the same page as different pages of duplicate content. My website, www.telikin.com currently has many versions of the home page. There is: www.telikin.com
Link Building | | Telikin
www.telikin.com/index
www.telikin.com/index.php
www.telikin.com/index.php/ All of these seem to be their own URLS, If i use the open site explorer tool to search for links pointing to these URLS, the www.telikin.com domain shows about 700 links, and the www.telikin.com/index.php shows about 63 links. My questions: Is my SEO suffering as a result of having these different pages? Does google realize that links pointing to www.telikin.com/index.php should count towards the seo of the www.telikin.com page? If this is hurting my company's website, how do I solve this problem? I am not versed as a webmaster, so I don't understand what the best course of action would be. Should I request that our webmaster somehow redirects all traffic from the other URLS to the main www.telikin.com URL? Should I be using canonical tags on all important pages? If i do, for example, use a canonical tag on the www.telikin.com page, will Google then know to send all link juice from the www.telikin.com/index.php page to the www.telikin.com page? We also have a Joomla blog that seems to duplicate pages many times over, but that is a problem that I will have to address another time. Any comments/suggestions are more than welcome. Thanks, Dave0 -
Are the following considered bad links for Link Building.
I've been doing a lot of research lately on link building & I've read a lot about having a diverse amount of links. Social links, Blog links, comment links, footer links, No follow links, Directory links as well as links with Anchor text URL match, brand match, partial brand match, non-descriptive, and with keywords. But there is a lot of talk of removing those links that look like a site that's built just for link building purposes.... but Directories are basically built for link building are they not? I'm wondering if anyone can tell me if they feel links from the following directories would be bad links. (Non-Paid) http://www.yoganza.com http://www.wkool.com http://www.wee-directory.com http://www.voicelet.com www.hotfrog.ca http://www.webdirectorieslist.com http://multiplelistingboard.com http://mozdirectory.net http://freedirectorypr.info Industry specific below: http://bestautolinks .com http://automotivedome.com http://automobilebuzz.com http://craigsautolinks.com Thanks so much for the help!
Link Building | | DCochrane0 -
Is there a crowd-sourced and up-to-date list of do-follow social bookmarking sites?
It's always frustrating to do a search for something like "do-follow social bookmarking sites", find a list and then find out through trial/error that many of them have switched to no-follow, are no longer around, etc. That's why I thought a crowd-sourced list might be more interesting and up-to-date. Any thoughts?
Link Building | | davegr0