Best way to block a sub-domain from being indexed
-
Hello,
The search engines have indexed a sub-domain I did not want indexed its on
old.domain.com and dev.domain.com - I was going to password them but is there a best practice way to block them.
My main domain default robots.txt says :-
Sitemap: http://www.domain.com/sitemap.xml
global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category//
Disallow: */trackback/
Disallow: */feed/
Disallow: /comments/
Disallow: /? -
Hi,
CleverPhD has some interesting ideas with robots.txt and Google Webmaster Tools, but simply password protecting all dev pages should keep pages out of Google's index. There's no best practice here, since a password wall will keep Googlebot out on its own.
To be doubly safe, you can also include a meta noindex tag on dev pages.
Keep in mind that once a page is in Google's index, it's going to take awhile for it to leave (unless you use CleverPhD's method). But, having a blank page in Google's index really isn't all that bad. It's there, but it won't rank for much.
Hope this helps,
Kristina
-
I've never tried a method like this - FreshFireOne, did you?
-
First and foremost when you finish all this - password protect your dev instances. A url will leak out eventually and then this happens. I know it is a PIA, but it is worth it.
To remove subdomains. Go into GWT and register the subdomains as separate websites in GWT. Create a robots.txt for each subdomain (not the one you mention, you need a robots that is specific to that subdomain that disallows all files. If you cant do that, have your subdomains include a noindex meta tag on all pages. You have to be careful with this as you do not want to push out your dev. robots.txt or the noindex meta tags to your production server, but it can be done. Talk to your devs. Then go into GWT and use the URL removal tool. Just leave it blank and it will remove the whole site.
Poof. Gone. You can then watch the GWT accounts. They will show errors for the dev site like "Severe health issues are found on your site - Some important page has been removed by request." This is a good error as it confirms that that subdomain is removed.
We actually used this not on a dev site but on our www1 server that was indexed. We use a load balancer with multiple copies of the site. www1 was completing with www. Using this above did the trick.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Question about getting domain name re-indexed
I recently swapped my domain from www.davescomputers.com to www.computer-help.com . Originally www.computer-help.com was 301 re-directing to www.davescomputers.com ...however my long term goal is to eventually rebrand my business so I decided to utilize the other domain by swapping the main domain. Is consistant blogging the best way to get Google to re-index the entire website? My focus has been quality posts and sharing them with vairus social profiles I created.
Intermediate & Advanced SEO | | DavidMolnar0 -
Why the archive sub pages are still indexed by Google?
Why the archive sub pages are still indexed by Google? I am using the WordPress SEO by Yoast, and selected the needed option to get these pages no-index in order to avoid the duplicate content.
Intermediate & Advanced SEO | | MichaelNewman1 -
301 many smaller domains to a new, large domain
Hi all, I have a question regarding permanently redirecting many small websites into one, large new one. During the past 9 years I have created many small websites, all focusing on hotel reservations in one specific city. This has served me beautifully in the past, but I have come to the conclusion that it is no longer a sustainable model and therefore I am in the process of creating one large, worldwide hotel reservations website. To not loose any benefit of my hard work the past 9 years, I want to permanently redirect the smaller websites to the correct section of my new website. I know that if it is only a few websites, that this strategy is perfectly acceptable, but since I am talking about 50 to 100 websites, I am not so sure and would like to have your input. Here is what I would like to do: (the domain names are not mine, just an example) Old website: londonhotels.com 301 to newdomain.com/london/ Old website: berlinhotels.com 301 to newdomain.com/berlin/ Old website: amsterdamhotels.com 301 to newdomain.com/amsterdam/ Etc., etc. My plan is to do this for 50 to 100 websites and would like to have your thoughts on if this is an acceptable strategy or not. Just to be clear, I am talking about redirecting only my websites that are in good standing, i.e. none of the websites I am thinking about 301'ing have been penalized. Thanks for your thoughts on this.
Intermediate & Advanced SEO | | tfbpa0 -
Are sub domains considered completely different than the root domain?
We have a project that is going to generate duplicate content. If we move the new content to a sub-domain (E.g. product.domain.com) will it still be considered duplicate content to the root domain? Or is it like having two completely different domains? Thanks!
Intermediate & Advanced SEO | | tripled5110 -
Redirect gateway domain to main domain?
We have following scenario: Our main website - www.esedirect.co.uk which gets a 1800 visits a day with around half of those from organic search. It's been around since 2004. Our original website - www.ese.co.uk which gets around 30 visits a day and really is nothing more than a doorway page with links to the above site and couple of other sites that belong to the same company. This is an old domain that's had content since 1997 and has good domain authority with some good links. We are considering doing a 301 redirect from www.ese.co.co.uk to www.esedirect.co.uk to redirect the link juice. I welcome opinions to any possible negative effects this could give and how beneficial doing this will be. Thanks, Lee
Intermediate & Advanced SEO | | ese0 -
Should I 301 a penalized domain to another domains subfolder?
I have a niche domain seems to have been hit by Penguin. It had very good rankings before the update, and I think at least a good part of the penalty might be due to overoptimized anchor text. So here is the question; If I decide to take this site down, should I 301 the entire domain to a relevant sub-folder of another site? i.e comtemporaryfurniture.com to domain.com/category/modern-furniture.html Will the penalty get passed onto the new domain? If the penalty is partly due to anchor text, then pointing it to another site's subfolder would mean the tartget URL has more varied anchor text and could boost rankings.
Intermediate & Advanced SEO | | inhouseseo0 -
Sub-domain and Sub-directory - Is there no difference
Hi, Last year Google announced that it will treat links from a sub-domain as internal links and not external - http://googlewebmastercentral.blogspot.in/2011/08/reorganizing-internal-vs-external.html. Does it means that now there is no difference between a sub-domain and a sub-directory and anyone of them can be used? Regards
Intermediate & Advanced SEO | | IM_Learner0 -
Domain certificate
Hello, I would like to know if there is any certificate we can buy to increase the Seo of my website. Thanks very much for your time
Intermediate & Advanced SEO | | Cartageno0