Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does redirected traffic still contribute to SEO?
My company recently rebranded with a new website (and comparatively lower authority). We've built a bunch of backlinks for our old website, which now redirect to the new one, and I was wondering if these still contribute to SEO? Is there a difference in the SEO effect of a direct link versus a redirected one?
Link Building | | Russell_Chua0 -
Results After a Disavow File Submission
Hi, I have a client who had a range of spammy links going to their website. The links were from exact match anchor text, from duplicate content and poor quality, irrelevant domains. They also had built a number of sites and linked to my client's site from each of the footers. All the domains they owned have been redirected to the site we're working on. We also disavowed all the spammy links going to the site. This was over a month ago and rankings haven't changed and in fact, have gone down slightly. Is there any other course of action anybody could suggest or should i wait longer in the hope the disavow is still yet to be accounted for? Any help I'd really, really appreciate! Thanks
Link Building | | SolveWebMedia0 -
Connecting Two Websites Without Upsetting Google
I have a question that I think I know the answer to, but want to cover my bases. We have a successful blog and are creating a second similar blog for a more specific type of reader. We want to backlink between content because it makes sense for the reader. Our fear is that we may come off spammy to Google by having so many backlinks. Is there a way we are able to tell Google these two sites are related so they don't penalize either site for anything?
Link Building | | HashtagJeff0 -
Does an external link to an image file on my site help my SEO?
I found a few websites that are displaying images from my site. Curiously, they have also linked these images to the source image file on my website. I found these because Moz showed these sites in my list of backlinks. I assume that means it impacts my Domain Authority. But does this actually help me in search engines since it's not linking to a page on my site?
Link Building | | AlexLenhoff0 -
Followed links from one of our biggest channel?
Hi everyone, I have a new client that's a huge company and has a disastrous ranking. It's a Human Resources company that among other things, they post job offers on their site in which people have to register to apply. As a channel to outreach more people, they are using a popular site for job offers on the country. There, the job position is described and they tell users to apply within this channel or on their own site. But there's no actual link to their site. Will Google find it spammy to post their site's link for that specific job offer on this channel on every job description? No competitor is doing it so it made me doubt a little bit. NOTE: The job offer channel doesn't allow us to "nofollow" a link. All the links are followed. Thanks to any suggestion.
Link Building | | guillermoga0 -
Are there benifits to aquiring domain names with keywords of a niche site and redirecting them to the niche site?
I f i purchased a domain with keyword another one of my sites targeted and redirected it to that site are there benefits in doing so? & as long as the domain contained the keywords would it matter if the domain name contained another word?. also would it be better to put just a few articles on the site and then point keywords in the article to the niche site?.
Link Building | | askshopper0 -
Google ranking penalty
Do you think that our websites involvement with LinkMonster.com link building program is the cause of our penalty by google in our ranking ?
Link Building | | websitedzyn0 -
301 to main page or deeper page for best 'link juice'.
I have two rafting websites. One ranks high for one set of keywords. The other ranks okay for a different set. I want to redirect the whole second site to the first to give the first more link juice and hopefully lift the first site even more in the rankings. Would it be better to link to a deeper page of the 1st site. I have a page that does well for some long tail keywords or would it be better to link to the main page. Or is it a bad idea over all.
Link Building | | tkobrien0