Disallow indexing of ALL subdomains
-
I'm using www.domain.com as my development hosting. Each website that i'm developing get's a temporary URL like this:
project1.domain.com
project2.domain.com
project3.domain.com
...Now i'd like to set that ALL these subdomains can not be indexed in Google. Now I manually have to do this for each subdomain's site, and when I go online I have to change the robots.txt again. So I would like to make things a bit easier for me.
Is this possible?
-
Hello there!
Like Wesley mentioned before the best way to avoid any "non-desired" crawling and indexing of your development or testing environment is by requiring authentication (whether with htaccess or your own programmed login/password screen).
Unfortunately sometimes there have been situations when search crawlers don't necessarily follow robots.txt directives. Additionally, beyond search engines you might want to protect your development or testing environment, making sure that only people with the required access to it can enter. Because of this, the best way to go is requiring authentication to access to your subdomains, not robots.txt.
I hope this help!
-
Nobody?
-
Thanks for the link, but I don't think that will solve my problem?
When Google wants to crawl project1.domain.com, he won't check the .htaccess of domain.com right? So then he'll just be crawling that subdomain?
-
You should edit your .htaccess file as is described here:
http://stackoverflow.com/questions/6738896/excluding-testing-subdomain-from-being-crawled-by-search-engines-w-svn-repositI hope i answered your question
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best practice to have gated white paper indexed by Google
Our main website white paper page has an image and brief description of the white paper. Once you click the white paper you are redirected to a form to access the gated white paper. Once you complete that form you are redirected to the white paper pdf which is housed on a subdomain/Hubspot. Because of this, I do not believe our website is getting "credit" for the keywords/content on these pages. Any suggestions on how we can allow the search engines to crawl this content while still keeping it gated? As I understand it a sub domain cannot hep or hurt (aside from critical crawler issues) the main domain. Thank you
On-Page Optimization | | NikCall0 -
Should I block google indexing "search.php"
My question is I have a search page on our website , you can search by date, number of people staying and so on, I am just wondering should block this in the robots.txt ? Because we have pretty URL'S already for searching by county and searching by towns. I cannot see any benefit of having e.g "search/search.php?sp_dateFrom=16%2F12%2F2015&sp_dateTo=23%2F12%2F2015&sec_drop%5B%5D=727&spesh_town_id=764&q=&occupants=5&bedrooms=3&submit=SEARCH#search" indexed. Would I be correct in doing this ?
On-Page Optimization | | McCaldin0 -
How to remove subdomains in a clean way?
Hello, I have a main domain example.com where I have my main content and then I created 3 subdomains one.example.com, two.example.com and three.example.com I think the low ranking of my subdomains is affecting the ranking of my main domain, the one I care the most. So, I decided to get rid of the subdomains. The thing is that only for one.example.com I could transfer the content to my main domain and create 301 redirects. For the other two subdomains I cannot integrate the content in my main domain as it doesn't make sense. Whats the cleanest way to make them dissapear? (just put a redirect to my main domain even if the content is not the same) or just change the robots to "noindex" and put a 404 page in the index of each subdomain. I want to use the way that will harm the least the performance with Google. Regards!
On-Page Optimization | | Gaolga0 -
Google Index Report
Hi, I have just checked my google webmaster tools account and viewed the index status of my website and it produced the attached graph, which show quite a big spike in indexing during July and August 2012. Does this look normal or does it reveal anything peculiar? We did have a new website launched in June 2012 and I re-submitted the sites URL's to google as part of the re-launch and so I am unsure if this may account for the spike. Any advice appreciated. Thanks indexing.png
On-Page Optimization | | UnderMe0 -
Is it good to have a subdomain with keyword?
Hi, I want to ask do you thing that it is good and necessary to have a subdomain with a keyword in it when the domain doesn't include it? f.e. you have a website named domain.com but there is no keyword in it. And if you add subdomain keyword.domain.com will this bring any benefit?
On-Page Optimization | | vladokan0 -
Index.php + external site added to end of URL
Good day, I have a domain http://www.ecofriendlylink.com. I am trying to resolve the Crawl Diagnostic errors on it. I have several Duplicate Page Content errors. Example 1: (The domain happynewyou is not mine, some Comments from them have been placed on my site. Ecoshop.php is a page on my site). URL: http://www.ecofriendlylink.com Duplicate Page Content: http://www.ecofriendlylink.com/www./happynewyou.com/ecoshop.php Referrer: None. Example 2: URL: http://ecofriendlylink.com/index.php Duplicate Page Content: http://www.ecofriendlylink.com/index.php http://www.ecofriendlylink.com/www./happynewyou.com/index.php Referrer: http://ecofriendlylink.com/ Example 3: is a different problem, but still a Dup Page Error. URL: http://ecofriendlylink.com/water.php Duplicate Page Content: http://www.ecofriendlylink.com/water.php Referrer: http://ecofriendlylink.com/ water.php is a page on my main domain. The www version and the non-www version, if this a problem and something I need to overcome? So please can you advise what I need to do to get rid of this strange external domain name + index.php (as per examples 1 + 2), and explain what I'm doing wrong with Ex 3. Thank you!
On-Page Optimization | | drkevinhogan0 -
Tags to index or to no index that is the Question
Sorry for the silly Shakespearean introduction but what are peoples opinions on leaving tag clouds to be indexed or no indexable by the search engines. I tend to leave them non indexable at the moment. I use wordpress for blogging as do most others.
On-Page Optimization | | onlinemediadirect0 -
Page Cache And Index
If you are browsing a site, what is the best way or programs to use to see if the page has been indexed and cached? Thanks
On-Page Optimization | | gregster10000