Does Google make continued attempts to crawl an old page one it has followed a 301 to the new page?
-
I am curious about this for a couple of reasons.
We have all dealt with a site who switched platforms and didn't plan properly and now have 1,000's of crawl errors. Many of the developers I have talked to have stated very clearly that the HTacccess file should not be used for 1,000's of singe redirects. I figured If I only needed them in their temporarily it wouldn't be an issue. I am curious if once Google follows a 301 from an old page to a new page, will they stop crawling the old page?
-
It will keep on crawling. I have seen 6 months to 1 year later they come back. Especially if there are links from other websites pointing to your old URLs.
Why they do repeat crawls on 301s and 404s for that matter is that they know how much the web changes and assume that there may have been a mistake and so want to double check. Also, you have to consider, they will not recrawl 1000s of pages all at once. They will probably pick up the most popular URLs first as they crawl them more often and then the less popular pages later.
IMHO. You will need to find a long term solution in place.
Here is the why you dont want to use htaccess
http://stackoverflow.com/questions/209209/htaccess-redirect-performance
One suggestion for Apache - RewriteMap
http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html#rewritemap
That or all the developers who are telling you not to use .htaccess should have at least 2 solutions to use instead that should work. Go back to them to solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Highly ranked pages to new domain?
Hi everyone! We are ranked #1 for about 30 product pages at www.oldsite.com/product1 and we are wanting to move about 30 of those pages to a new site www.newsite.com/product1 (new domain and hosting - which we own). What is the best way to do this? I'm confused if you recreate those pages on the new domain vs. ftp move them, 301 re-directs, etc. Looking for the things we must do and the sequence to do it all, etc. Thanks so much!
Intermediate & Advanced SEO | | Jamesmcd030 -
301 redirect to search results page?
Hi - we just launched our redesigned website. On the previous site, we had multiple .html pages that contained links to supporting pdf documentation. On this new site, we no longer have those .html landing pages containing the links. The question came up, should we do a search on our site to gather a single link that contains all pdf links from the previous site, and set up a redirect? It's my understanding that you wouldn't want google to index a search results page on your website. Example: old site had the link http://www.oldsite.com/technical-documents.html new site, to see those same links would be like: http://www.newsite.com/resources/search?View+Results=&f[]=categories%3A196
Intermediate & Advanced SEO | | Jenny10 -
Google Seeing 301 as 404
Hi all, We recently migrated a few small sites into one larger site and generally we had no problems. We read a lot of blogs before hand, 301'd the old links etc and we've been keeping an eye on any 404s. What we have found is that Webmaster is picking up quite a few 404s, yet when we investigate these 404s they are 301'd and work fine. This isn't for every url, but Google is finding more and I just want to catch any problems before they get out of hand. Is there any reason why Google would count a 301 as a 404? Thanks!
Intermediate & Advanced SEO | | HB170 -
Google still listing old domain
Hi We moved to a new domain back in March 2014 and redirected most pages with a 301 and submitted change of domain request through Google Webmaster tools. A couple of pages were left as 302 redirect as they had rubbish links pointing to them and we had previously had a penalty. Google was still indexing the old domain and our rankings hadn't recovered. Last month we took away the 302 redirects and just did a blanket 301 approach from old domain to new in the the thinking that as the penalty had been lifted from the old domain there was no harm in sending everything to new domain. Again, we submitted the change of domain in webmaster tools as the option was available to us but its been a couple of weeks now and the old domain is still indexed Am I missing something? I realise that the rankings may not have recovered partly due to the disavowing / disregarding of several links but am concerned this may be contributing
Intermediate & Advanced SEO | | Ham19790 -
Meta NOINDEX... how long before Google drops dupe pages?
Hi, I have a lot of near dupe content caused by URL params - so I have applied: How long will it take for this to take effect? It's been over a week now, I have done some removal with GWT removal tool, but still no major indexed pages dropped. Any ideas? Thanks, Ben
Intermediate & Advanced SEO | | bjs20100 -
Why do old URL format are still being crawled by Rogerbot?
Hi, In the early days of my blog, I used permalinks with the following format: http://www.mysitesamp.com/2009/02/04/heidi-cortez-photo-shoot/ I then decided to change this format using .htaccess to this format: http://www.mysitesamp.com//heidi-cortez-photo-shoot/ My question is, why do rogerbot still crawls my old URL format since these urls' no longer exists in my website or blog.
Intermediate & Advanced SEO | | Trigun0 -
301 from penalized domain to new domain
I have a client whose site isn't necessarily penalized since they still show for many terms in the SERPS, however at one point they did an xrummer blast of 13,000 links for two anchor texts they were trying to rank for. They have purchased a new domain and have gone white hat and want to 301 some of the old site to the new purely for the users sake so past visitors still find them at t the new location. Will creating 301 redirects pass on to the new domain any bad Karma from the old one in Google's eyes? Thanks for the help.
Intermediate & Advanced SEO | | JoshGill270 -
Rankings for Home vs. Internal Pages - Potential 301?
Hi everyone: A site I'm working with until recently was ranking page 1 for its primary keyword. For the last month, they've dropped to page 4. One thing we've noticed is that the page that is ranking is an internal page (http://www.example.com/keyword-string) and at this point, everything ranking above us is ranking based on the root domain (http://www.competitor.com). We've eliminated Panda, penalties, and any other obvious causes for the drop in rankings. We have similar or better page rank, external links, domain trust, etc. in comparison to the sites still ranking page 1. We think this may be part of our problem. Has anyone else dealt with this? What did you do to change it and how did it work? We're considering eliminating the existing internal page and 301'ing to the home page. The keyword in question is the core of the business, so this is a natural change, but we're loath to lose years of investment in promoting the internal page. Also, the site was originally optimized with the primary keyword throughout (appears in META tags, headers on multiple pages). How important is it to clear these out to make Google see the home page as most relevant? Thanks!!
Intermediate & Advanced SEO | | kdcomms0