Disallow indexing of ALL subdomains
-
I'm using www.domain.com as my development hosting. Each website that i'm developing get's a temporary URL like this:
project1.domain.com
project2.domain.com
project3.domain.com
...Now i'd like to set that ALL these subdomains can not be indexed in Google. Now I manually have to do this for each subdomain's site, and when I go online I have to change the robots.txt again. So I would like to make things a bit easier for me.
Is this possible?
-
Hello there!
Like Wesley mentioned before the best way to avoid any "non-desired" crawling and indexing of your development or testing environment is by requiring authentication (whether with htaccess or your own programmed login/password screen).
Unfortunately sometimes there have been situations when search crawlers don't necessarily follow robots.txt directives. Additionally, beyond search engines you might want to protect your development or testing environment, making sure that only people with the required access to it can enter. Because of this, the best way to go is requiring authentication to access to your subdomains, not robots.txt.
I hope this help!
-
Nobody?
-
Thanks for the link, but I don't think that will solve my problem?
When Google wants to crawl project1.domain.com, he won't check the .htaccess of domain.com right? So then he'll just be crawling that subdomain?
-
You should edit your .htaccess file as is described here:
http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-repositI hope i answered your question
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best use of canonical ref on home (default/index) page?
Moz reports show duplicate content for my www.domain.com and www.domain.com/default.html How do I format the canonical ref to put on the default.html page?
On-Page Optimization | | bhsiao0 -
Should I "No Index" Certain Pages On My Site?
I have some pages on my site that don't really have any content other than some iframes that are embedded from another site. I thought it would be best to tag the page with a no-index so that search engines would leave the page alone since it has no content as far as the search engine can tell (but does provide value to my site visitors). Is this the proper approach or does it do more harm than good?
On-Page Optimization | | Kyle Eaves0 -
Blocking Subdomain from Google Crawl and Index
Hey everybody, how is it going? I have a simple question, that i need answered. I have a main domain, lets call it domain.com. Recently our company will launch a series of promotions for which we will use cname subdomains, i.e try.domain.com, or buy.domain.com. They will serve a commercial objective, nothing more. What is the best way to block such domains from being indexed in Google, also from counting as a subdomain from the domain.com. Robots.txt, No-follow, etc? Hope to hear from you, Best Regards,
On-Page Optimization | | JesusD3 -
New jobboard: Can redirecting folder (site.com/jobboard) to subdomain (jobboard.site.com) hurt SEO?
Hi there, I'm planning to implement a jobboard on my website which needs to be installed on a subdomain (jobboard.site.com) but I'd really like to use site.com/jobboard for promoting this jobboard (jobboard collects external industry jobs). Are there any possible disadvantages when I set up a 301 redirect from jobboard.site.com to site.com/jobboard? Also: What if I want to move this jobboard to a unique domain one day (e.g. jobboard-industry-xy.com), Would that be tricky (as I'd basically have to redirect the folder-to-subdomain redirect to an external domain and therefore get a folder-to-subdomain-to-external-domain redirect...)? Cheers, Thomas
On-Page Optimization | | stl990 -
Why isn't our site being shown on the first page of Google for a query using the exact domain, when its pages are indeed indexed by Google
When I type our domain.com as a query into Google, I only see one of our pages on the homepage, and it's in 4th position. It seems though, that all pages of the site are indexed by google when I type in the query "site:domain.com". There was an issue at the site launch, where the robots.txt file was left active for around two weeks. Would this have been responsible for the fact that another domain ranks #1 when we type in our own domain? It has been around a couple of months now since the site was launched. Thanks in advance.
On-Page Optimization | | featherseo0 -
Duplicate page content & title for www.mydomain.com and www.mydomain.com/index.php?
Hi, First post so please be gentle! My Crawl Diagnostics Summary is showing an error relating to duplicate page content and duplicate page title for www.mydomain.com and www.mydomain.com/index.php which are, in my view, the same thing/page? Could anyone shed any light please? Thanks Carl
On-Page Optimization | | Carl2870 -
Sitemap generator and /index.htm
I use DMX Zone sitemap generator in Dreamweaver. When generating the sitemap, it includes both http://www.domain.com and http://www.domain.com/index.htm. Should the /index.htm be deleted from the sitemap, or will the 301 redirect take care of that? Best,
On-Page Optimization | | ChristopherGlaeser
Christopher0 -
3 Different Home Page URL's Being Indexed?
Hello Everyone! I own a dog supplies eCom site on the x-cart platform. I recently upgraded to 4.4 version about 3 weeks ago and am noticing 3 different home page URL's getting indexed and ranked: /
On-Page Optimization | | k9byron
/home.php
/home.php?cat= I dont know why this is happening and I dont claim to be an expert SEO but know this cant be good! I am seeing high rankings on certain terms for all 3 URL's. Has anyone seen this before and can anyone give me any feedback on this and how it may be effecting my sites ranking in the future? Thanks in advance!
Byron-0
