Development Website Duplicate Content Issue

[SO_UK]

Hi,

We launched a client's website around 7th January 2013 (http://rollerbannerscheap.co.uk), we originally constructed the website on a development domain (http://dev.rollerbannerscheap.co.uk) which was active for around 6-8 months (the dev site was unblocked from search engines for the first 3-4 months, but then blocked again) before we migrated dev --> live.

In late Jan 2013 changed the robots.txt file to allow search engines to index the website. A week later I accidentally logged into the DEV website and also changed the robots.txt file to allow the search engines to index it.

This obviously caused a duplicate content issue as both sites were identical. I realised what I had done a couple of days later and blocked the dev site from the search engines with the robots.txt file.

Most of the pages from the dev site had been de-indexed from Google apart from 3, the home page (dev.rollerbannerscheap.co.uk, and two blog pages). The live site has 184 pages indexed in Google. So I thought the last 3 dev pages would disappear after a few weeks.

I checked back late February and the 3 dev site pages were still indexed in Google. I decided to 301 redirect the dev site to the live site to tell Google to rank the live site and to ignore the dev site content. I also checked the robots.txt file on the dev site and this was blocking search engines too. But still the dev site is being found in Google wherever the live site should be found.

When I do find the dev site in Google it displays this;

Roller Banners Cheap » admin   dev.rollerbannerscheap.co.uk/ A description for this result is not available because of this site's robots.txt – learn more.   This is really affecting our clients SEO plan and we can't seem to remove the dev site or rank the live site in Google.

In GWT I have tried to remove the sub domain.

When I visit remove URLs, I enter dev.rollerbannerscheap.co.uk but then it displays the URL as http://www.rollerbannerscheap.co.uk/dev.rollerbannerscheap.co.uk.

I want to remove a sub domain not a page.

Can anyone help please?

[TomRayner]

Hi Lewis

To be honest, you've done everything absolutely right.  The only other thing I could suggest would be to add to the head of those pages, if you haven't already.

But from everything you've said and having tested it myself, it doesn't look as though Google now has any chance of reading duplicate content from the dev sub-domain.

Have you received a warning in Google webmasters?  Have you submitted a reconsideration request if you think you have been penalised?

There shouldn't be any duplicate content issue now that is affecting the live site's ability to rank. Panda refreshes may take a while, if indeed you have been hit at all, but you've done everything here spot on in my eyes.

With that in mind, I'd focus on building quality content and links to your live site, in order to try and rank it.  I think you just need to show a bit of blind faith in Google to sort this out themselves.

[Mike.Goracke]

Hi Lewis,

If you haven't already, you need to verify your subdomain (dev.rollerbannerscheap.co.uk/) in Google Webmaster Tools. Once verified, you can then request removal of your entire site, which in this case is your subdomain.

You will need to follow Google's Requirements for removing content to help you through the rest of the process.

Hope this helps.

Mike