Block a sub-domain from being indexed

[kchandler]

This is a pretty quick and simple (i'm hoping) question. What is the best way to completely block a sub domain from getting indexed from all search engines?

One item i cannot use is the meta "no follow" tag.

Thanks! - Kyle

[X-X]

Keep in mind that Google Index's everything that it can crawl.  Even if you put a block in the robots.txt they will probably crawl it.  You can require a password to that subdomain and keep big G out.  This is easy to do if you have a site with cpanel access.  Just go to manage permissions, and password protect that director with a .htaccess pw.

[bloggidy]

The robots.txt file just tells the bots you would "prefer" they don't index but there is nothing to prevent them from indexing.The only sure way to do this is to restrict access to the sub-domain for everyone and require some sort of authentication. If they don't have access they can't index.

[Oversteer]

In subdomain.example.com/robots.txt add the statements:

User-agent: *
Disallow: /

Warning: Be absolutely certain that the above statements are not included in your example.com/robots.txt file or you'll kill your site.

[john4math]

Each subdomain may have its own robots.txt file.  So for that subdomain, you can put:

User-agent: *
Disallow: /

In the robots.txt, and that should do it.

Please note that disallowing pages in robots.txt will not necessarily mean they won't appear on search result pages.... if people link to pages that are disallowed on that subdomain, they can still appear in SERPs.  I had this happen with a few pages, which leads to funny listings in the SERPs because Google has to guess what the page title and description of the page should be, since it's not allowed to read the page.  The meta noindex tag is the way to go if you want to be really sure the page doesn't appear in the SERPs.  If you use that, don't disallow the page.  Here's a recent SEOMoz post about it: http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts

[kchandler]

That was going to be my assumption but i wasn't 100% sure how they worked with sub domains. Are you able to supply a little more information on implementation? It is extremely important that it only blocks: sub.domain.com and not domain.com

[kennyrowe]

Block using your robots.txt page.