Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Any benefits to having Wikipedia links now they are 'no-followed' (apart from traffic and natural link prof.)
I see that Wikipedia outbound links are all no-followed, is there any benefit (aside from the traffic) for having links here now ? For example is their co-citation and co-occurance benefits. I know there is without the links since from seeing previous Moz content about this saying Google getting good at connecting brand/s and topic mentions on a page (without any links) so appreciate Wikipedia is still good for that sort of thing. And a no-followed link is obviously good for the potential traffic. But is there any additional SEO benefit to having a no followed link on a wikipedia entry/stub too ? (aside from its contribution to your no-followed links which in turn contribute to a natural looking link profile) Cheers Dan
Link Building | | Dan-Lawrence0 -
Google Link Disavow and sites with extremely heavy but poor link metrics
Need help/opinions from the SEO's out there. I am working with a site that in the past hired an SEO company out of India. Over the course of their time together, this company submitted the clients url to tens of thousands of link exchanges and directories. Around the time of Penguin, the owners told me their inquiries dried up. Literally that same week they had a new website launched (designed by a local competitor) that really butchered the site. They were convinced the reduction in traffic/inquiries was due to the new site but I am convinced otherwise (suspect Penguin). Not only does the site need to be re-structured but their link portfolio needs to be diversified. Now on the plus side, the company caters and sponsors a number of events that earns them organic, relevant links. These are overshadowed by the tons of poor, irrelevant ones though. I read a few posts recently on the Google Link Disavow tool and to proceed slowly due to the unknown nature surrounding it. I have a good idea of what links are problematic and which ones are well served. However, the number of problematic links in my estimation is quite high (thousands). I am very hesitant to dive into Google Disavow and submit such a large number of requests. What are your thoughts? How would you proceed?
Link Building | | mattylac1 -
How to have a good position in Google bot ?
Hello,
Link Building | | Hamza9
my website is not appearing in the first positions in Google, it appears in the 1st position only if you type it address, but for example if you type a keyword that can lead to it, It will NOT appears, what can I do ?0 -
Are the following considered bad links for Link Building.
I've been doing a lot of research lately on link building & I've read a lot about having a diverse amount of links. Social links, Blog links, comment links, footer links, No follow links, Directory links as well as links with Anchor text URL match, brand match, partial brand match, non-descriptive, and with keywords. But there is a lot of talk of removing those links that look like a site that's built just for link building purposes.... but Directories are basically built for link building are they not? I'm wondering if anyone can tell me if they feel links from the following directories would be bad links. (Non-Paid) http://www.yoganza.com http://www.wkool.com http://www.wee-directory.com http://www.voicelet.com www.hotfrog.ca http://www.webdirectorieslist.com http://multiplelistingboard.com http://mozdirectory.net http://freedirectorypr.info Industry specific below: http://bestautolinks .com http://automotivedome.com http://automobilebuzz.com http://craigsautolinks.com Thanks so much for the help!
Link Building | | DCochrane0 -
Why does my site only have 4 internal followed links/internal links?
When using Open Site Explorer my site is winning in many categories but lacking in others as compared to competitors. I intend to improve on this but am a little confused by the extremely low number of "internal followed links" and "internal links" at just 4. My competitors have much more than that. Does this mean links to the root domain? Total internal links? I have many internal links so I'm having a hard time figuring this one out. What four internal links are getting noticed? I just went through and changed all the links that went to index.html to link to the root domain at www.njlimocarservice.com as I just found a Q&A that indicates this was better. So as of Wed at 5:45 PM all my links to the homepage went to index.html and not the root which is no longer the case. Not sure if this matters. At the same time I also just put a 301 from index.html to redirect to www.njlimocarservice.com. Any help appreciated! h9QuF.png
Link Building | | kabledesigns0 -
Can No Follow Links Hurt Rankings?
Thanks to sites like SEOmoz and good folks like yourselves I was able to understand enough about SEO to know that the company we outsourced the SEO to were doing at best a sloppy job. I have seen that the link profile of site is totally unnatural in terms of factors such as relevancy and anchor text but also that around 95% of the 800 links are no follow. My question is whether these low quality no follow links causing unnatural anchor text ratios and poor link relevancy would hurt our rankings? I am basically wondering if I have to now go and try to have all those links removed somehow to avoid any problems going forward?
Link Building | | Luia0 -
Product Giveaways and No-Follow
If a blogger is given a product from a company for a giveaway contest on their blog, does Google require the link to the company have a no-follow. This seems like a gray area but I can't find any guidelines put out by Google about this.
Link Building | | SparkplugDigital0 -
No Follow Blogs. Do they help with keyword density?
I know no follow blogs don't pass page rank or link juice, but do they aid in help with making your site know for a specific keyword when you use anchor text?
Link Building | | photoseo10