How to handle potentially thousands (50k+) of 301 redirects following a major site replacement
-
We are looking for the very best way of handling potentially thousands (50k+) of 301 redirects following
a major site replacement and I mean total replacement.Things you should know
Existing domain has 17 years history with Google but rankings have suffered over the past year and yes we know why. (and the bitch is we paid a good sized SEO company for that ineffective and destructive work)
The URL structure of the new site is completely different and SEO friendly URL's rule. This means that there will be many thousands of historical URL's (mainly dynamic ones) that will attract 404 errors as they will not exist anymore. Most are product profile pages and the God Google has indexed them all. There are also many links to them out there.
The new site is fully SEO optimised and is passing all tests so far - however there is a way to go yet.So here are my thoughts on the possible ways of meeting our need,
1: Create 301 redirects for each an every page in the .htaccess file that would be one huge .htaccess file 50,000 lines plus - I am worried about effect on site speed.
2: Create 301 redirects for each and every unused folder, and wildcard the file names, this would be a single redirect for each file in each folder to a single redirect page
so the 404 issue is overcome but the user doesn't open the precise page they are after.
3: Write some code to create a hard copy 301 index.php file for each and every folder that is to be replaced.
4: Write code to create a hard copy 301 .php file for each and every page that is to be replaced.
5: We could just let the pages all die and list them with Google to advise of their death.
6: We could have the redirect managed by a database rather than .htaccess or single redirect files. Probably the most challenging thing will be to load the data in the first place, but I assume this could be done programatically - especially if the new URL can be inferred from the old.Many be I am missing another, simpler approach - please discuss
-
Sorry to hear of your woes.
Depending on the structure of the URLS you could create some simple pattern matches rules within .htaccess? If you could a few dozen rules could handle many thousands of redirects. If there isn't any easily identifiable pattern to match then a DB will, indeed, be your best option.
One of the web devs I used to work with (who was considerably smarter than me) faced a similar issue (with a 'mere' 10k+ redirects) and used some Ruby on Rails middleware as a redirector: This may have been the solution he used:
https://github.com/vigetlabs/redirector
I hope that helps.
I hope you're able to get this sorted without too much pain. Good Luck!
-
Thank for the very quick response - you have picked my favourite solution. It will be interesting to hear other views and comments.
-
Hi,
1. Usually won't work and with 50k extra rules in your htaccess file it will for sure slow down the site as for every request to your server it has to go through the htaccess file.
For now I would recommend going with 6. with the information that you've provided. By doing it like this you can do a very quick check on your database and also in the request have the user send to the right page.
Martijn.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How long will old pages stay in Google's cache index. We have a new site that is two months old but we are seeing old pages even though we used 301 redirects.
Two months ago we launched a new website (same domain) and implemented 301 re-directs for all of the pages. Two months later we are still seeing old pages in Google's cache index. So how long should I tell the client this should take for them all to be removed in search?
Intermediate & Advanced SEO | | Liamis0 -
How do you handle a site with inherited negative links, but no penalty?
I'm trying to rank a new client for various key phrases that contain "it support." The problem is that about 100 of their 180 total referring domains have links that include "it support" (usually as partial match, or if exact then for uninteresting terms with low traffic), mostly on quite low quality directories. So, no penalty, and not much exact match I'm worried about, but I'm concerned that there's too high a percentage overall of partial match or simpy "it support"-based links for me to continue building keyword-optimized links to try and rank for the much harder terms we need to rank for... Despite the large number of low quality directories, a disavowal does not seem like a good idea since there is no penalty, but how does one avoid being handicapped by such bad links that came before one's time?
Intermediate & Advanced SEO | | zakkyg0 -
Question about moving content from one site to another without a 301
I could use a second opinion about moving content from some inactive sites to my main site. Once upon a time, we had a handful of geotargeted websites set up targeting various cities that we serve. This was in addition to our main site, which was mostly targeted to our primary office and ranked great for those keywords. Our main site has plenty of authority, has been around for ages, etc. We built out these geo-targeted sites with some good landing pages and kept them active with regularly scheduled blog posts which were unique and either interesting or helpful. Although we had a little success with these, we eventually saw the light and realized that our main site was strong enough to rank for these cities as well, which made life a whole lot easier, not to mention a lot less spammy. We've got some good content on these other sites that I'd like to use on our main site, especially the blog posts. Now that I've got it through my head that there's no such thing as a duplicate content penalty, I understand that I could just start moving this content over so long as I put a 301 redirect in place where the content used to be on these old sites. Which leads me to my question. Our SEO was careful not to have these other websites pointing to our main site to avoid looking like we were trying to do something shady from a link building perspective. His concern is that these redirects would undermine that effort and having a bunch of redirects from a half dozen sites could end up hurting us somehow. Do you think that is the case? What he is suggesting we do is remove all of the content that we'd like to use and use Webmaster Tools to request that this content be removed from the index. Then, after the sites have been recrawled, we'll check for ourselves to confirm they've been removed and proceed with using the content however we'd like. Thoughts?
Intermediate & Advanced SEO | | LeeAbrahamson0 -
Merging 4 websites into one for a new site release (301 question)
Hi guys and girls, I have a client that has 4 very outdated websites with about 50 pages on each. They are made up like: 1 brand group and 3 for each individual key service they offer, so let's call them: brand.com (A) brand-service-1.com (B) brand-service-2.com (C) brand-service-3.com (D) We've rebuilt the main site and aggregated all the content from the others (99% re-written). Am I correct in thinking the process for the new lauch would be: 1. Launch the new site on brand.com (A) and 301 all the old brand.com (A) pages to the related pages on the new site. 2. Redirect the other websites (B,C,D) on a domain level to the new site on the brand.com (A) domain. 3. Clean up the old URL's, sitemaps, errors in Google WMT Is this right? Anything I missed/better practices? I was also wondering if I should redirect B,C,D in stages, or use page level redirects.
Intermediate & Advanced SEO | | shloy23-2945840 -
301 Redirect and Webmaster Central
I've been working on removing canonical issues. My host is Apache. Is this the correct code for my htaccess? RewriteEngine On
Intermediate & Advanced SEO | | spkcp111
RewriteCond %{HTTP_HOST} ^luckygemstones.com$ [NC]
RewriteRule ^(.*)$ http://www.luckygemstones.com/$1 [R=301,L] SECOND!!! I have two websites under Google's Webmaster Central; http://luckygemstones.com which gets NO 404 soft errors... AND http://www.luckygemstones.com which has 247 soft 404 errors... I think I should DELETE the http://luckygemstones.com site from Webmaster Central--the 301 redirect handles the"www" thing. Is this correct? I hate to hose things (even worse?) Help! Kathleen0 -
Site revamp for neglected site - modifying site structure, URLs and content - is there an optimal approach?
A site I'm involved with, www.organicguide.com, was at one stage (long ago) performing reasonably well in the search engines. It was ranking highly for several keywords. The site has been neglected for some considerable period of time. A new group of people are interested in revamping the site, updating content, removing some of the existing content, and generally refreshing the site entirely. In order to go forward with the site, significant changes need to be made. This will likely involve moving the entire site across to wordpress. The directory software (edirectory.com) currently being used has not been designed with SEO in mind and as a result numerous similar pages of directory listings (all with similar titles and descriptions) are in google's results, albeit with very weak PA. After reading many of the articles/blog posts here I realize that a significant revamp and some serious SEO work is needed. So, I've joined this community to learn from those more experienced. Apart from doing 301 redirects for pages that we need to retain, is there any optimal way of removing/repairing the current URL structure as the site gets updated? Also, is it better to make changes all at once or is an iterative approach preferred? Many thanks in advance for any responses/advice offered. Cheers MacRobbo
Intermediate & Advanced SEO | | macrobbo0 -
Aged domain and 301 redirect? (11 year old domain)
Hey everyone, I'm about to launch a new website for an accounting firm. They currently have a website, which has an 11 year old domain. They are doing very well locally for SEO, and i'm guessing it's because of the aged domain, as their website is very badly built, and contains almost no content. They would like to launch the new site with a simpler, easier to remember domain. If i launch the new site, point the aged domain using a 301 redirect, and do redirects for all of the old pages to the newer versions of them, is there a chance the company will lose their current SEO rankings? Thanks!
Intermediate & Advanced SEO | | RCDesign740 -
BIG CHANGE - 301 Main site to new domain
Hi Guys, Were wondering what to do about our main domain name, we were ranking quite high for our main keyword and before Christmas our site dropped to 10th and we have been there for a while - last week our site dropped again onto the second page. The worrying thing is now our main domain name is now ranking 1 place above another domain name that we don't really use but its an exact match domain name for our target keyword. This exact match domain has hardly any links pointing to it and it currently has a 22 domain authority. We are wondering if we 301 our main site to this exact match domain would it rank higher than the top of the 2nd page where we are now for our main domain. Cheers.
Intermediate & Advanced SEO | | ScottBaxterWW1