Culling 99% of a website's pages. Will this cause irreparable damage?
-
I have a large travel site that has over 140,000 pages. The problem I have is that the majority of pages are filled with dupe content. When Panda came in, our rankings were obliterated, so I am trying to isolate the unique content on the site and go forward with that.
The problem is, the site has been going for over 10 years, with every man and his dog copying content from it. It seems that our travel guides have been largely left untouched and are the only unique content that I can find. We have 1000 travel guides in total.
My first question is, would reducing 140,000 pages to just 1,000 ruin the site's authority in any way?
The site does use internal linking within these pages, so culling them will remove thousands of internal links throughout the site.
Also, am I right in saying that the link juice should now move to the more important pages with unique content, if redirects are set up correctly?
And finally, how would you go about redirecting all theses pages? I will be culling a huge amount of hotel pages, would you consider redirecting all of these to the generic hotels page of the site?
Thanks for your time, I know this is quite a long one,
Nick
-
Thank you all for the positive feedback.
Lately I have made the time for SEOmoz Q&A as I have been doing various SEO research and these boards can be a great way to stretch thought processes.
-
Seriously, Ryan is always ALL OVER Seomoz comments with good feedback
-
Just figured out how to do this, I'm new to SEOMoz Q&A, thanks for the nudge! Ryan certainly deserves it!
-
I do hope Ryan gets "Good Answer" and/or "Endorsed Answer" for this... hint, hint
-
Your understanding is correct.
Google does not care how many directories appear in a URL. The two URLs you offered as an example are viewed equally by Google. What's important is how many clicks it takes users to access those links.
-
Hi Ryan,
Sorry for not getting back to you straight away, I've been in meetings all day.
You've given me some excellent ideas again!!
Just to clarify, the old URL's are in the following format:
www.url.com/resort_hotels/hotels_in_rome.asp
I am aiming to use the following structure for the new website:
or
I was wondering if you knew from a search engine perspective, which URL is the better option. From a user perspective, I would assume the second.
I am operating under the assumption that Google rates a URL's importance by the number of clicks it is from the homepage and not the number of directories (www.url.com/.../.../...) within the URL?
If this is the case I will probably go for the second URL structure, but place links higher up the hierarchical structure of the site for the more important locations.
Unfortunately, the landing pages for the cars and flights house exactly the same content with just the location text tweaked. There is nothing else unique on these pages, which is why I find myself with no other option but to get rid of them.
I really like your idea of testing landing pages for a specific area. This may be a good way to go, but creating two paragraphs of text for both the flight and car hire pages is not an option at this time. With 40,000 locations we’d need to produce 160,000 paragraphs of unique text, which would cost around $400,000, may be slightly less with bulk discounting.
If I was to spend that much money on content writing, I would probably expand the hotel side of the site as this is most profitable. But my priority after the launch of the new site is an extensive link building campaign to assist the transition.
Thanks so much again for your help Ryan, you're a star!
Did you know whether Google rates a URL's importance by the number of clicks it is from the homepage and not the number of directories (www.url.com/.../.../...) within the URL? It is really important that I find this one out!
Take care buddy,
Nick
-
Nick,
Sounds like you have a good strategy. I only have two additional items to share based on your latest reply.
www.url.com/resort_hotels/hotels_in_rome.asp
That url seems a bit spammy to me. Mentioning "hotels" twice is something I would avoid. I would consider something along the lines of the below options instead:
www.url.com/resorts/hotels_in_rome
www.url.com/resort_hotels/rome
I also wanted to talk about the landing pages for cars and air travel once more. Before directing all your current pages to a generic page I would take a look at the existing 140 pages and ask once again, do any of the pages have anything that is unique which can be used for the location based car and air landing pages?
Your plans are to develop these pages with quality content over time, which is great. I hate the idea of having establishing pages for each area, pulling back to having one generic page, then expanding again to location-based pages.
If you sincerely intend to develop these pages on a reasonable time period, I would suggest establishing one page for each location even if it was thin on content to start with. Driving directions, local driving laws, testimonials, anything that can be used as a starting point to hold your footing would be preferred.
If you do pull back to a generic "car rentals" page, I have two ideas. Build out your location landing page for one area such as London. Closely watch your conversion rates on users on the London page versus the generic page. If there is a significant difference, it may help speed up your transition. If you realize you are losing $$ every day you don't have those pages, then perhaps you can hire additional help to speed up the process.
The final idea would be to build country-based landing pages for car rentals as an stop-gap measure. Your Milan, Rome, etc pages could all direct to "Cars Italy" and "Air Italy".
There are tons of choices on the internet for travel providers. You have an extremely well established user base. My top concern for any migration is to maintain all my existing relationships. Some travel sites do great with a single landing page for air/cars/hotels. It sounds like your site has catered to clients in a specific way, and I would be sensitive to maintaining your current user experience.
One last idea that just came to me. After the migration poll users for feedback. Take surveys, offer discounts, generate hype but engage users because they will offer a different point of view which you may not have considered.
-
Ryan, you have given me some excellent ideas here and a great overall structure to make the transition between sites. I can't thank you enough for your help. I will certainly consult an SEO before proceeding with anything, but your insight has given me a lot to think about.
With regards to the sites current pages, the majority of locations only have 3 pages; Hotels, Car Hire & Flights. It is the amount of locations covered that make the site so expansive.
So with Hotels being our biggest earner, my idea going forward was to:
-
Use the travel guide's unique content for the hotel landing pages, i.e. [Hotels in Rome]
-
Redirect all of the old Car & Air location pages to the new website’s generic Car Hire & Flights pages.
This would mean that there wouldn’t be any location-based pages for Flights and Car Hire. The idea would be to build these up gradually as it would take some time and money to add the unique content required.
- From every hotel landing page we would use anchor text to promote the generic Flights and Car Hire pages. For example, [Buy Cheap Flights] or [Cheap Car Hire]
This additional anchor text should help our external link building and the generic Flights and Car Hire pages would house a search form for users to search any location.
So essentially, the majority of the site would be made up of Hotel landing pages, until we began building the site further.
I can see that your main concern is that the correct redirects are in place.
The site currently has the following URL structures, with locations for each:
Apart from the sitemaps, each have locations with them, for example:
www.url.com/resort_hotels/hotels_in_rome.asp
So my idea is to:
1) Redirect all “resort_hotels” URL’s to their relevant hotel page on the new website, for example,
www.url.com/resort_hotels/hotels_in_rome.asp
will go to the “Hotels in Rome” page on the new website.
- The rest of the pages will be redirected to the home page for their category, for example,
will go to “Flights” home page on the new website; and,
will go to “Car Hire” home page on the new website, etc.
Unless there is something really wrong with this strategy, or you have any instant criticism, I would like to thank you for your help again and ask that if you need anything, please don’t hesitate to drop me a message on here. You have given up enough your time and I’m more than grateful.
Kind Regards,
Nick
[I am using my work’s account, which is why I am displayed as Steve]
-
-
The transition I mentioned would allow for a smoother migration process rather then a "cold turkey" switch from the old site to the new site. You clearly recognize the end goal is to create your new site and delete the old site. The good news is that change does not have to happen over night.
You can build out your new site completely and go live with it. At that point you would update any external links you control along with your advertisements, signatures, etc. You would also want to reach out to partners and any sites with links that you can influence. Update those links so they point to your new pages.
The final step is the redirection of your 140k page old site to the appropriate pages on the new site. Clearly you wish to begin with the most prominent pages such as your landing pages along with any important pages such as "Contact Us", your reservation system, etc.
The next step would be applying your redirect rules to the remaining pages. Extensive testing will be required.
You should set up GA or another tracking tool to monitor your old site. You will want to closely monitor activity for quite some time. Specifically look for any issues with 404s and multiple redirects.
With respect to your anchor text, I suspect it was used to sculpt your site so your link value was focused on a particular page for each topic. When you have 140 pages on a given topic, you can pursue an incredible amount of longtail phrases. Now I suspect you may have 4 pages for each area: Rome, Rome by Air, Rome by Car, and Rome hotels. If that is the case your future anchor text linking will be a lot more straight forward.
I want to say "I wouldn't be concerned about the anchor text" but you have a major project ahead of you, you are highly dependent on SEO and there are many opportunities for something to go wrong. In that context, I would share the anchor text would be on the list of things to think about, but the proper redirects is a much larger concern.
A final thought I would offer: this is all high level, generic advice. I would recommend hiring a SEO who could offer a proper evaluation of your site along with a migration plan. Once the change has been completed and tested, you should gain many advantages with your new site. Hopefully they will offset any loss from the migration. Once you are confident in your new site, I would recommend a SEO campaign promoting your new site.
-
Hi again Ryan,
All the URL's are currently coded as .asp (www.url.com/Rome.asp) and we aim to build the new site with user friendly permalinks (www.url.com/Rome). So in answer to your question, yes, the sites could co-exist.
I'd hadn't thought of doing it this way, what a great idea.
With regards to the site's internal linking structure, I'm probably not explaining myself correctly. I understand that all of the site's juice needs to be recycled, but I'm now thinking that on many of the 120,000 pages there are links with anchor text to other relevant parts of the site, will removing these links, because there are so many of them, ruin the site's authority.
In addition, I would be really interested to hear your ideas on staging a transition.
I can't thank you enough for this Ryan, my head's spinning at the moment!
-
You are on the right track. The link value from your existing pages must be saved.
Prior to offering a further reply I would like to ask a couple questions:
-
how are your currently URLs coded? As an older site I presume your page URLs end in .asp?
-
will your new design also be in asp?
What I am trying to determine is, will the new site require new URLs. If your current page is /rome.asp and the new page will be /rome.php then the URL will change so both your new site and old site can co-exist at the same domain. This process will be helpful for staging a transition.
PS. My recommendation for URLs would be to use friendly URLs which do not show an extension (i.e. /rome) but that is not the present focus.
-
-
Thanks for a swift answer Ryan, very helpful indeed!
Put simply, the site is split into three key areas, Hotels, Flights & Car Rentals, each with about 40,000 pages each. The problem is that each of these pages uses a generic paragraph or two that is more or less the same, but tweaked slightly to update the location in question. For example,
"Our goal is to provide the best choice of hotels in Rome."...
"Our goal is to provide the best choice of hotels in Barcelona."...
Obviously Google sees this as duplicate content and rightly so, but other than rewriting 120,000 pages of content, I can't see an alternative to the problem, other than to remove the pages in question.
The site has so many quality links going into it, from authorities all over the web; it would be a shame to waste this juice on pages that are getting penalised.
The travel guide areas are all unique; there is a single guide for each of the 1000 destinations. For example,
http://www.url.com/guides/rome
My idea was to use this unique content to promote our hotel pages, for example,
http://www.url.com/hotels-in-rome
This page would have the unique travel content from that area plus a list of the hotels we have available in Rome.
Every other duped page on the site relating to "Rome", does have "Rome" in its URL, so a regex expression could be used to redirect all "Rome" themed pages to the "Hotels in Rome" page that would house the unique content.
All other pages that did not have unique content written about them could be redirected to the generic Hotels, Flights or Car Rentals pages as they all have either “hotels”, “flights” or “car-rental” in their URL.
The site is over 10 years old, is written in .asp and is managed with a bespoke piece of software created specifically for the site itself.
However, this doesn't really matter as we’re having the website redesigned at the same time as removing the dupe content and it will be built to our own specification.
My idea is to begin building up these locations from when the redesign goes live. This way I could keep a track of our content as it expands.
My main worry is that the culling of these pages, will remove 99% of our internal linking structure. And I'm wondering if removing this will dramatically reduce the authority of the site. However, at this point I’m struggling to see another option.
Sorry for the length of this reply, any ideas would be welcome, I just thought it would be best if you knew a bit more of the background.
Thanks again Ryan!
-
Hi Steve.
I would suggest taking a good look at your content pages. I understand having dupe pages, but you are suggesting a possible 140:1 ratio which is....wow.
As I am sure you are aware, there is not going to be any quick and easy fix. Here are some initial thoughts:
The first step I would take is to look for ANY commonalities between pages you can grab. For example, you mention 1000 travel guides. Are all the travel areas unique? For example, is there only one guide for Rome? If so, do all the duplicate Rome pages have "Rome" in the URL? If so, you can consider adding a regex expression to your htaccess file (presuming you are on an apache server) which could cover your 301s.
Is there any unique code on the pages which are common for a given guide? Using the above example, do all the "Rome" pages have "Rome" in the page title? If so, you could possibly update all the "Rome" pages by adding the correct canonical to the page's meta information. At this point it would truly depend on how your site is coded. Do you have a CMS? In what language are your pages written?
The bottom line, there is simply too much value in those pages to discard them. They each need to be properly 301'd as the preferred method. The 301s really need to be handled with general expressions which cover a large number of pages at once. You cannot use individual redirects even if you wanted to as it would cripple your web server.
I would not redirect all the pages to a single home page unless every other opportunity was completely explored.
-
Wow that's a big, bold move! I don't know how to answer it but if I were you I'd wait until you get a few, nice and comprehensive answers on here before doing anything to drastic. Either that or use a private Q&A question to SEOmoz staff if you have any points spare to do so. With such a large change, you want to ensure you're doing it right.
I'll be interested to see the answers you get for this.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Over-optimizing Internal Linking: Is this real and, if so, what's the happy medium?
I have heard a lot about having a solid internal linking structure so that Google can easily discover pages and understand your page hierarchies and correlations and equity can be passed. Often, it's mentioned that it's good to have optimized anchor text, but not too optimized. You hear a lot of warnings about how over-optimization can be perceived as spammy: https://neilpatel.com/blog/avoid-over-optimizing/ But you also see posts and news like this saying that the internal link over-optimization warnings are unfounded or outdated:
Intermediate & Advanced SEO | | SearchStan
https://www.seroundtable.com/google-no-internal-linking-overoptimization-penalty-27092.html So what's the tea? Is internal linking overoptimization a myth? If it's true, what's the tipping point? Does it have to be super invasive and keyword stuffy to negatively impact rankings? Or does simple light optimization of internal links on every page trigger this?1 -
How necessary is it to disavow links in 2017? Doesn't Google's algorithm take care of determining what it will count or not?
Hi All, So this is a obvious question now. We can see sudden fall or rise of rankings; heavy fluctuations. New backlinks are contributing enough. Google claims it'll take care of any low quality backlinks without passing pagerank to website. Other end we can many scenarios where websites improved ranking and out of penalty using disavow tool. Google's statement and Disavow tool, both are opposite concepts. So when some unknown low quality backlinks are pointing and been increasing to a website? What's the ideal measure to be taken?
Intermediate & Advanced SEO | | vtmoz0 -
Can't generate a sitemap with all my pages
I am trying to generate a site map for my site nationalcurrencyvalues.com but all the tools I have tried don't get all my 70000 html pages... I have found that the one at check-domains.com crawls all my pages but when it writes the xml file most of them are gone... seemingly randomly. I have used this same site before and it worked without a problem. Can anyone help me understand why this is or point me to a utility that will map all of the pages? Kindly, Greg
Intermediate & Advanced SEO | | Banknotes0 -
Website Isn't Ranking & I'm Not Sure Why Based On The Data
Hi Moz Community,
Intermediate & Advanced SEO | | ErrickG
I am having an issue that has been killing me for some time and I could really use another opinion. One of my client’s websites hasn't been ranking for some time and I can't put my finger on it. There are no issues showing up in the webmaster tools. If you compare the site with the tops ranking sites for the websites number one keyword, the website is just as good as everyone else. My clients website is the first one on the left in the attachment. We have better quality content but instead of showing up on page 1,2,3 the site is on page 21. I am just at a lost. Anyone have any thoughts outside looking in. Thanks,
Errick rrLJZ2G0 -
Site's disappearnce in web rankings
I'm currently doing some work on a website: http://www.abetterdriveway.com.au. Upon starting, I detected a lot of spammy links going to this website and sort to remove them before submitting a disavow report. A few months later, this site completely disappeared in the rankings, with all keywords suddenly not ranked. I realised that the test website (which was put up to view before the new site went live) was still up on another URL and Google was suddenly ranking that site instead. Hence, I ensured that test site was completely removed. 3 weeks later however, the site (www.abetterdriveway.com.au) still remains unranked for its keywords. Upon checking Web Master Tools, I cannot see anything that stands out. There is no manual action or crawling issues that I can detect. Would anyone know the reason for this persistent disappearance? Is it something I will just have to wait out until ranking results come back, or is there something I am missing? Help here would be much appreciated.
Intermediate & Advanced SEO | | Gavo0 -
Will I lose traffic from Google for re-directing a page?
I’m currently planning to a retire a discontinued product and put a 301 redirect to a related product (although not identical). The thing is, I’m still getting significant traffic from people searching for the old product by name. Would Google send this traffic to the new pages via the re-direct? Is Google likely to display the new page in place of the old page for similar queries or will it serve other content? I’d like to answer this question so that I can decide between the two following approaches: 1) Retiring the old page immediately and putting a 301 redirect to the new related pages. This will have the advantage of transferring the value of any link signals / referring traffic. Traffic will also land on the new pages directly without having to click through from another page. We would have a dynamic message telling users that the old product had been retired depending on whether they had visited out site before. 2) Keep the old product pages temporarily so that we don’t lose the traffic from the search engines. We would then change the old pages to advise users that the old product was now retired, but that we have other products that might solve their problems. When this organic traffic decreases over time, then we will proceed with the re-direct as above. I am worried though that the old product pages might outrank the new product pages. I’d really appreciate some advice with this. I’ve been reading lots of articles, but it seems like there are different opinions on this. I understand that I will lose between 10% - 15% of page rank as per the Matt Cutts video.
Intermediate & Advanced SEO | | RG_SEO0 -
What's the Best Host For WordPress sites
Our site has gone down twice in a week...hosted by Fat Cow. So we're going to switch hosts this week. We currently have 2 WP sites on a Fat Cow VPS. 8 GB file size and 2 GB data transfer monthly. We use a CDN and video hosting company (Wistia) so the file sizes are small. I've contacted several hosts and narrowed it down to WP Engine, Rack Space and A Small Orange. I care about fast page load time (1 second), 99.999% up-time and great support. Price is a secondary concern. I'm leaning towards WP Engine, but wanted to ask Moz community before making a decision. Any other hosting companies I should call?
Intermediate & Advanced SEO | | Branden_S0 -
Include Cross Domain Canonical URL's in Sitemap - Yes or No?
I have several sites that have cross domain canonical tags setup on similar pages. I am unsure if these pages that are canonicalized to a different domain should be included in the sitemap. My first thought is no, because I should only include pages in the sitemap that I want indexed. On the other hand, if I include ALL pages on my site in the sitemap, once Google gets to a page that has a cross domain canonical tag, I'm assuming it will just note that and determine if the canonicalized page is the better version. I have yet to see any errors in GWT about this. I have seen errors where I included a 301 redirect in my sitemap file. I suspect its ok, but to me, it seems that Google would rather not find these URL's in a sitemap, have to crawl them time and time again to determine if they are the best page, even though I'm indicating that this page has a similar page that I'd rather have indexed.
Intermediate & Advanced SEO | | WEB-IRS0