Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are backlinks from the Episode Description field on Spotify "follow" or "no follow"?
Are backlinks from the Episode Description field on Spotify "follow" or "no follow"?!
Link Building | | FLLocal0 -
How to create link from google redirect?
I have seen redirect links from google but not sure how to create one. Please guide if anyone knows the answer. Example: https://images.google.cv/url?q=https://moz.com
Link Building | | Melissacarter3 -
In Google search console all of sudden a lot of backlinks have disappear at "Link To Your Site"
Hi after update in Google search console yesterday a lot of backlinks in "Link To Your Site" have disappear, before we had 43x domains now we have 12x domains linking to the site. I have checked the sites where we had link do-follow before and they are OK. Can you please tell me the reason what has gone wrong. And also please guide me in fixing the issue , Hope to hear from you soon..! URL: https://www.finanstopp.no/
Link Building | | heleneolsen3 -
How to index Backlinks Fast in Google!
I Made few backlinks with my keyword but they are not getting index and it has been for 15 days. Do you have any idea that will help me out to solve this problem and also to index back links quickly.
Link Building | | rohitdeveloper0 -
Googles stance on Back Links via a Badge/Form
Hey guys, Does anyone know what Google's stance is on backlinks that come via a form, WordPress theme or badge. For example if I offer website security and provide badges for websites that are malware clean (with a back link to my website) and 100 websites sign up to my website will this be deemed as bad practice in Google's eyes? Also if I create a free WordPress theme with a backlink to myself? The second question sounds like I'm providing content for a link which seems okay but the first one can go either way. Thanks
Link Building | | conversiontactics0 -
Do inbound links to download .exe files improve SEO?
The site, (http://www.bartenderbarcodesoftware.com/) sells barcode software. They also provide free printer drivers on their site. During a recent domain change (before I started helping them), they began looking at their inbound links and found quite a few external sites linking to these download files. However, when someone clicks on the link on one of these sites (i.e. http://vvs-motor.com.ua/barcode-printer-drivers) it immediately begins downloading the file (i.e. http://www.seagullscientific.com/downloads/drivers/archive/7.2/7.2.4/Unimark_7.2.4.exe) without taking the visitor to the site. The owner would greatly prefer this traffic to engage with the site and is considering 301 redirecting those download links to an actual webpage. Question: Would this look suspicious to Google? Would this have a positive or negative effect on SEO? The last factor to consider is that many of these sites will likely just recreate the list based on the new links at some future point. Thanks for any help!
Link Building | | motosantiago0 -
Value of Link Redirect from Google News
Some sites are linking to us with the URL from Google News instead of using a direct link. For example: "http://news.google.com/news/url?sa=t&fd=R&usg=ghjgjhggjhgkjghjg&url=http://www.oursite.com/awesome-news-010123/" Does this pass the same value as a normal link? I asked the publisher to replace the link with a normal link but they pushed back saying that the news URL is better.
Link Building | | ProjectLabs0 -
How does Google interpret articles or prepositions in languages where it's attached to the (key)word?
Hi, All! This is for any foreign language SEOs where articles or prepostitions such as "the" "to" "in" or anything else are actually part of the word they are modifying and not a separate word, as in English: How does Google understand those words on-page and in anchor text? If you want to optimize for the word "house", and your content/anchor text says "the house" or "in the house" (again, all one word) - what does Google count that as? Does it count toward "house"? Does it count toward "in the house" only? Does it count toward "house" but not as much as if you had just put "house"? I end up sometimes writing slightly grammatically-off content because I want to optimize for the keyphrase - but is that necessary? Obviously different languages might be different, but you can probably project a little from one to the others. Thanks in advance!
Link Building | | debi_zyx0