The robots.txt file just tells the bots you would "prefer" they don't index but there is nothing to prevent them from indexing.The only sure way to do this is to restrict access to the sub-domain for everyone and require some sort of authentication. If they don't have access they can't index.
Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Best posts made by bloggidy
-
RE: Block a sub-domain from being indexed
-
RE: SEO from Godaddy How Good is it?
I personally use GoDaddy for purchasing domains but I do not use any of their add-ons. I would not recommend paying for their SEO tools. Submitting to any search engine is a free process and one that is easy to do.
For Bing go to: http://www.bing.com/toolbox/webmaster/
For Google go to: http://www.google.com/webmasters/
Setting up an account and doing so will also allow you to become familiar with their tools and reports that they provide. It is a great way to learn how and when they are indexing your site.
One additional thing to consider is to create a sitemap that dynamically gets updated as your site does. Submit this sitemap to the search engines and then it will improve the crawling efficiency and accuracy of the search engines. The individual responsible for your web development should be able to set this up for you.
-
RE: What is the best method to block a sub-domain, e.g. staging.domain.com/ from getting indexed?
Use an .htaccess file to only allow from certain ip addresses or ranges.
Here is an article describing how: http://www.kirupa.com/html5/htaccess_tricks.htm
-
RE: What is the best method to block a sub-domain, e.g. staging.domain.com/ from getting indexed?
.htaccess files can be placed at any directory level of a site so you can do it for just the subdomain or even just a directory of a domain.