Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I want to know about the Impact of Do-follow backlinks and No-follow backlinks for increasing DA.
Have Do-follow backlinks and No-Follow backlinks same value to increase Domain Authority in Moz? Recently, I have created 290 profile backlinks which was do-follow. But from yesterday I have seen all backlinks are no-follow now according to the decision of sites' owners. My site's DA has already increased up to 35. Will this DA fall gradually from now? Apart from it, if I have created same backlinks for my another site; will my DA be 35? Please try to clarify it. My site is: Homeworkpaper.net Thanks. Waiting for your valuable answer.
Link Building | | darrellpc1 -
How much time does it takes for a backlink to get registered in Google? My basketball anime website has been live since start of this week and Google has not registered any backlink.
So my website https://www.animedhfa.com/basketball-anime/ about basketball Anime has shown no changes in DA or backlinks. It has been 5 days perhaps since it went live.
Link Building | | ahmad74750 -
Redirecting dofollow, high-domain-authority links from one site to another: good idea?
I have two sites that offer help for freelance writers. The first gets almost no search traffic and is very low priority for me. The second is my main priority and gets substantial search traffic. The low-priority site has a significant number of dofollow backlinks from high-domain-authority sites. Is it a good idea to 301 redirect these links to similar/related posts on my high-priority site? Would this potentially boost the SEO of my high-priority site? Thanks for any help!
Link Building | | John88990 -
In Google search console all of sudden a lot of backlinks have disappear at "Link To Your Site"
Hi after update in Google search console yesterday a lot of backlinks in "Link To Your Site" have disappear, before we had 43x domains now we have 12x domains linking to the site. I have checked the sites where we had link do-follow before and they are OK. Can you please tell me the reason what has gone wrong. And also please guide me in fixing the issue , Hope to hear from you soon..! URL: https://www.finanstopp.no/
Link Building | | heleneolsen3 -
Are stackoverflow links follow or nofollow?
I've tried to find the answer to this question myself, but I've found differing opinions. The conclusion I've come to is that Stackoverflow allows follow links when the user that posts the link has sufficient reputation or if the link receives user validation. Has anyone else here used Stackoverflow.com that knows the answer to this question? 🙂
Link Building | | RG_SEO0 -
Spammy no follow backlinks - what should I do?
There is a individual/company that is creating lots of spam sites in my niche. It doesn't seem that they are trying to rank for any keywords, so I'm a little confused as to their purpose, but here's what they're doing. They scrape the top 10 Google results for each keyword and create a page - so if the niche is "widgets", they scrape the top 10 ranked sites for 'widgets', 'blue widgets', 'red widgets', and so on. A 'results' page is created for each keyword, which is linked from a home page. The results page always contains 10 websites - text is the site metadescription or similar. Each website gets a nofollow link back with the page title used as the anchor text. The host sites all have the keyword in the domain name - e.g www.widgetsxyz.com. The sites are thrown together but interestingly have a very crappy but individual logo (like 90's clipart). The host sites all have high DA/PA due to some very extensive link spam pointing to the sites, with very targeted keywords. There is seemingly no purpose to these sites that I can see - no other 'followed' links on the page or site. Whoever is doing this is churning out tens, if not hundreds of these sites. Any ideas what might be going on here, and whether I should be disavowing these sites (even though no follow). The fear is that they could switch the links to do follow, and get all the major players penalised in one fell swoop. ???
Link Building | | johnohara0 -
Best SEO practice to redirect affiliate link
Hello, I got an affiliate program on my website, that redirects the affiliate link to the main site like: site.com/ads/aff_code/ -> site.com/ (The redirect is done using a 301 status code.) On the redirect process the site stores a cookie to track the affiliate sale. Will Google and others SE follow this permanent redirect, transferring the relevance of this affiliate link to my main site? In other words, if an affiliate does something wrong (like spams), does the bad reputation will be transferred to my main site? Is there a better way to do that from a SEO standpoint? Thanks,
Link Building | | darkmediagroup0 -
Google Disavow File Update
Is there any specific format to update the Disavow file? Also if I submitted the file a months ago, and need to update it now... should I leave the old 'excluded domains' or should I remove them? Lets say this is what I have: How would you update it? #explanation from to Google went here... and ended here.
Link Building | | dhidalgo1
"domain:exampledomainalreadysubmitted1.com"
"domain:exampledomainalreadysubmitted2.com"
"domain:exampledomainalreadysubmitted3.com" Thanks for your input0