Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How Does Google Treat Scapper Sites?
I have seen several sites that have data or some type of information about a website. Often the site appears to be trying show the value of a site. How does Google treat the links this scrapper type sites give? Should these types of links be ignored or should they be disavowed? Here is an example: http://www.sitetracer.com/www.tourexperience.com
Link Building | | Firestarter-SEO0 -
301 redirect of PR 6 domain to PR 1 site, Will it improve my site's seo?
I have a pagerank 1 website but i recently purchased one old pagerank 6 domain, if I 301 redirect this pr6 domain to my currrent pr1 site, will it will increase my site's pr and improve seo ?
Link Building | | IndiaFPS0 -
Do follow blog roll links
Hi I found out that someone is linking to my site from their blog roll (which is a related theme), which I am flattered. She is linking to lots of others including quality blogs. I ran a back link report and I'm on 250 of her pages with a dofollow link. The anchor text is not a "money" keyword. In fact it's got nothing to do with what I do! The domain is DA38. Now, here's the questions going through my mind! Do I ask her to make them no follow? Do I ask her to remove them? Do I ask her to brand them all instead of irrelevant anchor text? Do I just disavow them? They are like having sitewide links on a site Any thoughts? SMP
Link Building | | intmarkacademy0 -
How to make google index the page title has changed ?
Google has indexed all my websites, but I have to change some title in the website but when checking google does not update the change.
Link Building | | tuananh0688
Please help me .
My site : http://blog.addme.vn0 -
Did Google Panda and Penguins have any effect on Outbound Links?
I am just a beginner in SEO. But has any of the changes recently implemented by google affected out bound links? Meaning if a website now has a good number of outbound links to strong websites and pages could that Translate to better rankings?
Link Building | | sherohass0 -
Slapped by Google update
Last couple of days we read things about Google algorithm and some things about companies or people doing "negative seo".. At first, nothing was wrong... if you play the game by the rules you won't get penalized you think... right? Today I checked rankings for some sites we're optimizing, and I fell of my chair! A site that was building better rankings week by week just dropped heavily and we have no clue if it's the algorithm update... Attached there are some screenshots, the first one showing the ranking for one of the most important keywords for that business, the others are link profiles & moztrust & rank for that root domain. As you can see, link profiles are much stronger that competitors, but that important keyword is dropping in rankings... We still have to do some on page optimizations (double title tags & some some canonical tags that have to be added, but that can't be the reason for this major drop?? Or am I missing something??? Any ideas? Are External Followed links growing to fast? (The blue line in screenshots is the site we're working on) rankings-may.jpg moztrust.jpg followed-links.jpg
Link Building | | nvs.nim0 -
How long google & roger need to notice backlinks
Hello, It is three weeks I have begun link building to a client website through partnerships and neither Google (link:mysite.com) nor Roger are seing any of these backlinks Any idea what is the time commonly needed for small and new websites (no trafic at the moment) to see domain authority growing ? Loïc
Link Building | | mandinga0 -
Retain keywords with redirect
I manage a site the sells product A and product B (both major keywords for us). If I buy the url www.productB.com and redirect it to the original website, would the original website gain any benefits in search engines? Would the benefits of having a url with the keyword be lessened because there isn't content hosted there, only a redirect to the original site?
Link Building | | livethesource0