Should I issue a change of address for my demo site which has been indexed?

[iam-sold]

The web design company tasked with building a company website have a demo site which reflects the live site - as the site is constantly being developed and needs to be tested by numerous uses, removing the demo site is not an option.

The demo site is a subdomain of the live site: demo.mysite.com

The demo site has been indexed by Google as the web design company didn't add a robots.txt file disallowing all. The demo site was indexed 3 years before I started working at the company.

My first action was to add a robots.txt file disallowing all, I then added a noindex and nofollow meta tag into the header of all files on the demo area - this has prevented any new pages from being indexed but the original pages of the site are still in Google index.

Do you think I should add the demo subdomain (demo.mysite.com) into Google Webmaster and issue a change of address pointing to the actual website (www.mysite.com) or is this going to cause problems?

All thoughts, opinions and recommendations are very much appreciated.

[iam-sold]

Hi Paul,

Some great advice.

In the turn of the New Year when we have more resources we are going to take action following your plan - it is a lot more work but makes perfect sense.

Again, thank you for pointing out that I need to remove the robots.txt block so that the page commands can be discovered - I will get onto this straight away.

Thank you once again.

Lloyd

[ThompsonPaul]

You've got a couple of issues here, Lloyd, that aren't going to be solved by your proposed solution. The biggest one is - after three years, you have a huge amount of dev site pages indexed, and it is quite likely those pages now have at least some incoming links. If you just de-index those pages, that link and indexing equity will be thrown away, and users will continue to arrive on the "wrong" pages for quite some time.

In my experience, to do this correctly, you've got a bit more work ahead of you.

1. Move the dev site to a new subdomain.
2. As soon as it's moved, add a met-robots declaration to the dev site's htaccess file to automatically add no-index/no-follow to the headers of every page. If you wish, at this point, you can also add a block in robots.txt.
3. Optionally, (and most preferably) password-protect the dev site by setting up a password in htaccess, assuming there aren't a huge number of people who need to have dev site access.
4. 301-redirect all the existing (old) dev site's pages to the equivalent pages on the main Domain. This can be done with a single rewrite rule in htaccess.
5. Removing the robots.txt block from the current dev subdomain would also be a good idea.
6. Submit the main section pages of the Dev site using the Fetch as Google/Bing bot tools in the respective Webmaster Tools to try to speed up the discovery of the 301 redirects. This should help get the redirects crawled and indexed sooner, and therefore hopefully drop out of the index a bit quicker.

The main reason for doing it this way is to preserve all the link/indexing equity that has built up over the three years. In addition, it ensures any incoming dev links or users with bookmarked links will immediately end up on the correct pages. Remember, a notice to users on the dev domain page will mean the user will have to figure out how to find the correct page on the "real" domain - not a user experience you want to inflict on your customer. By using the above solution, they can't get to the "wrong" pages in the first place.

Does that all makes sense? More work, but for a much cleaner solution, both for the user and for the search engines.

Oh, and if you decide to go the easy way and leave the current dev site in place as no-indexed, you're going to have to remove the robots.txt block. No-index and robots.txt block CANNOT be used together if the pages are already indexed. The robots block tells the search engines not to crawl the pages, which means they can never the discover the no-index commands and so will never drop the pages from the index. (That's why your existing pages haven't dropped out.)

Lemme know what you think;

Paul

[iam-sold]

Great, thank you for your advice Jeff.

From past experiences, how long does it usually take for Google to de-index pages that have the noindex and nofollow tag in the header?

[customerparadigm.com]

Llyod-

I wouldn't recommend the change of address feature. It's meant to be used when you are migrating your main website to a new domain. Your primary (live) domain is intact; you just want to no-index / no-follow your demo site.

The change of address will also only be effective for 180 days, too.

Google recently changed their change of address tool, and Barry Schwartz of Search Engine Land walks through it:
http://searchengineland.com/google-revamps-change-address-tool-within-webmaster-tools-205829

The big takeaway:
"Don't use this tool unless you are moving your primary website presence to a new address."

I hope this helps!
-- Jeff

[iam-sold]

Hi Jeff,

Sorry for taking so long to reply and thank you for taking the time to respond.

1. It isn't possible to password protect the demo sites for numerous.

2. The demo and live site are hosted using a load server which has multiple IP addresses, all of which change from week-to-week meaning that we cant restrict the IP addresses using the htaccess file.

3. I like the idea of adding a banner to the top of the demo site letting people know that they are not on the live site, we could also include a link through to the live site - which help usability.

Can I ask why you don't think I should issue a change of address?

Thanks,

Lloyd

[customerparadigm.com]

This happens a lot more than you might imagine.  We've even seen times where end users are placing orders on test sites because they're able to find out-of-stock items from an out-of-date demo site.

If you need to have the demo. store up and running so people can test it, I don't think you need to tell Google about the change of address.  I think that the no-index / no-follow status of the site and the robots.txt should prevent Google from indexing it.

That said, there are a few more ways to protect the demo site:

1. You can password protect it, using an .htaccess file.  This might not be ideal, because testing might prevent you from being able to see the whole site properly on different devices.

2. You could IP restrict the site to a small number if IP addresses, using your .htaccess file.  In this way, only people who are whitelisted are able to do this.  You could even get fancier, and have a form that allows testers to be able to add their IP address to the "allowed" list from a browser.  Anyone else would / could be redirected to the normal site.

3. I'd also recommend putting a banner at the top of the demo site, letting people know that any orders placed there are just for testing and won't be fulfilled.

I hope this helps!

-- Jeff