Duplicate site (disaster recovery) being crawled and creating two indexed search results
-
I have a primary domain, toptable.co.uk, and a disaster recovery site for this primary domain named uk-www.gtm.opentable.com. In the event of a disaster, toptable.co.uk would get CNAMEd (DNS alias) to the .gtm site. Naturally the .gtm disaster recover domian is an exact match to the toptable.co.uk domain.
Unfortunately, Google has crawled the uk-www.gtm.opentable site, and it's showing up in search results. In most cases the gtm urls don't get redirected to toptable they actually appear as an entirely separate domain to the user. The strong feeling is that this duplicate content is hurting toptable.co.uk, especially as .gtm.ot is part of the .opentable.com domain which has significant authority. So we need a way of stopping Google from crawling gtm.
There seem to be two potential fixes. Which is best for this case?
- use the robots.txt to block Google from crawling the .gtm site
2) canonicalize the the gtm urls to toptable.co.uk
In general Google seems to recommend a canonical change but in this special case it seems robot.txt change could be best.
Thanks in advance to the SEOmoz community!
-
It's a little tricky. While Andrea is right about Robots.txt - it's not great for removal once pages/domains are indexed, you can block the sub-domain with robots.txt and then request removal in Google Webmaster Tools (you need to create a separate account for the sub-domain itself). That's often the fastest way to remove something from the index, and if it has no search value, I might go that route. Just proceed with caution - it's a delicate procedure.
Doing 1-to-1 canonicalization or adding 301 redirects may be the next strongest signal (NOINDEX is a bit weaker, IMO). However, Google will have to re-crawl the sub-domain to do that, so you'll need to keep the paths open.
-
First, if the pages are already indexed then a robots.txt won't make them go away. A meta tag no index on the pages is the better solution. This allows search engines to "read" you page, see the no index tag and then work to remove the pages from index. A robots.txt doesn't necessarily accomplish the same result.
-
If you can do a 1-to-1 page canonicalization (each page on .co.uk is canonicaled to the equivalent page on the .com) then I would do that.
Otherwise, I would noindex the backup site.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have a same paragraph appearing on two webpages of my site!
is it gonna affect rankings if so what should be done thanks
Intermediate & Advanced SEO | | Sam09schulz0 -
Bolded words in search results
are those synonyms or semantically related keywords ? Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
Is there a way to no index no follow sections on a page to avoid duplicative text issues?
I'm working on an event-related site where every blog post starts with an introductory header about the event and then a Call To Action at the end which gives info about the Registration Deadline. I'm wondering if there is something we can and should do to avoid duplicative content penalties. Should these go in a widget or is there some way to No Index, No Follow a section of text? Thanks!
Intermediate & Advanced SEO | | Spiral_Marketing0 -
Combining two existing sites into a single magento install
Hi, We run an online beauty ecommerce store and recently acquired one of our competitors. Their site runs on magento also, and they sell 70% the same product as us. We plan to merge the new site into our existing magento install but keep both sites looking exactly as they do now with different themes, different product names, product descriptions, product prices, category structures etc. In theory the customer would have no idea both sites from the same magento, they will look just as they do now. My question is, will google possibly slap the SERP's of either sites because we have combined them onto the same server and same magento install, even though nothing on either site actually changed on the front end. Both sites already have the same ownership information on the domain WHOIS, and a quick company search would reveal that we legally own both businesses under the same company. So it's not something we are trying to hide, we are open about it, and plan to continue running both sites long term, with each site being targeted to a slightly difference audience, with 30% different products at different price points. Has anyone done this before? Was there any SEO risks or SERP drops? Would love some advice on this matter before we make the move, the possible blow back is way too massive to do it without firm advice saying the risk is very low. Brad.
Intermediate & Advanced SEO | | rec1230 -
How to make sure dev site is not index in wordpress and how would it be affected?
hi guys! I'm currently having a dev version of my site (dev.website.com) that once everything is done i would move the dev to the public domain (website.com). But since is a total duplicate content of my real site would it affect the seo? if so, i tried setting the reading privacy in wordpress so google would not index it but im afraid when i live it in the future and revert the setting back to normal it would affect the site seo. any opinion and suggestion on this?
Intermediate & Advanced SEO | | andrewwatson920 -
Sitelinks in non-brand based organic search results
Hi all, I have a question for everyone. Sitelinks have been around for a while now & I've always seen them when the search is for a brand's name. However, today, when looking at the rankings for one of the campaigns we manage, we noticed there were sitelinks in the number #1 & #2 positions in Google (Australia) for the search term "Dance Costumes". Whilst both the companies have Dance Costumes in their title, so do all the other results & so I don't see why it warrants the sites to be relevant via their brand name.
Intermediate & Advanced SEO | | KBB_Digital
Note: The results are organic results, not paid results (where you can add sitelinks). Firstly, has anyone seen this before (screenshot attached)?
And secondly, is there markup/schema that allows you to do this (none that I know of)? danceCostumes-sitelinks.png0 -
Merging two existing company sites into one
Hi Moz community, I have recently started a new job for a Fire & Security company in the UK to help with their non existent SEO efforts. Currently they have two separate websites. One of the websites is for their services and the other website is for their eCommerce store selling fire alarm equipment etc. The eCommerce store is higher up in the SERPs and overall has a lot more links. It also uses a better branded domain name. As I have never attempted such a project I have a few questions. The current eCommerce store is hosted and maintained by another web company which uses their bespoke CMS. What I want to do is take the service website and merge it with the ecommerce domain, however the service site runs on wordpress, which I want to continue for its flexibility. The service page wants to be the new homepage with a link on it to go to the store. I just cant get my head around the whole operation so if anyone could give me some advice to point me in the right direction that would be great. Thanks
Intermediate & Advanced SEO | | BradNichol0 -
Best way to duplicate a wordpress site for staging purposes?
I want to make some changes to my Wordpress site, and want to somehow set up a live staging area. Does anyone know of a good way to do this? I want all of the same content there I just want to be able to make changes to it and try it all out before going live. Any thoughts on this? Also I want to be sure the staging site doesn't get indexed since it will be a complete duplicate of my existing site. Thanks!
Intermediate & Advanced SEO | | NoahsDad0