Block a sub-domain from being indexed
-
This is a pretty quick and simple (i'm hoping) question. What is the best way to completely block a sub domain from getting indexed from all search engines?
One item i cannot use is the meta "no follow" tag.
Thanks! - Kyle
-
Keep in mind that Google Index's everything that it can crawl. Even if you put a block in the robots.txt they will probably crawl it. You can require a password to that subdomain and keep big G out. This is easy to do if you have a site with cpanel access. Just go to manage permissions, and password protect that director with a .htaccess pw.
-
The robots.txt file just tells the bots you would "prefer" they don't index but there is nothing to prevent them from indexing.The only sure way to do this is to restrict access to the sub-domain for everyone and require some sort of authentication. If they don't have access they can't index.
-
In subdomain.example.com/robots.txt add the statements:
User-agent: *
Disallow: /Warning: Be absolutely certain that the above statements are not included in your example.com/robots.txt file or you'll kill your site.
-
Each subdomain may have its own robots.txt file. So for that subdomain, you can put:
User-agent: * Disallow: /
In the robots.txt, and that should do it.
Please note that disallowing pages in robots.txt will not necessarily mean they won't appear on search result pages.... if people link to pages that are disallowed on that subdomain, they can still appear in SERPs. I had this happen with a few pages, which leads to funny listings in the SERPs because Google has to guess what the page title and description of the page should be, since it's not allowed to read the page. The meta noindex tag is the way to go if you want to be really sure the page doesn't appear in the SERPs. If you use that, don't disallow the page. Here's a recent SEOMoz post about it: http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts
-
That was going to be my assumption but i wasn't 100% sure how they worked with sub domains. Are you able to supply a little more information on implementation? It is extremely important that it only blocks: sub.domain.com and not domain.com
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does having a sub-domain on a different server affect SEO?
I'm working with a company that has a hard-coded website on the root domain, and then a WordPress blog on a subdomain on a separate server. We're planning on implementing a hub and spoke model for their content, hosting the main hubs on the root domain and the linked articles on the blog. Is having the blog on a different server going to hinder our SEO efforts?
Technical SEO | | KaraParlin0 -
Sudden Indexation of "Index of /wp-content/uploads/"
Hi all, I have suddenly noticed a massive jump in indexed pages. After performing a "site:" search, it was revealed that the sudden jump was due to the indexation of many pages beginning with the serp title "Index of /wp-content/uploads/" for many uploaded pieces of content & plugins. This has appeared approximately one month after switching to https. I have also noticed a decline in Bing rankings. Does anyone know what is causing/how to fix this? To be clear, these pages are **not **normal /wp-content/uploads/ but rather "index of" pages, being included in Google. Thank you.
Technical SEO | | Tom3_150 -
Old Redirected Domain is replacing my current domain on SERPs
Hello everyone, All of a sudden a 2 year old redirected domain is replacing my current domain for 2 weeks now, my site is apitus.com and my old domain is aptitus.pe (the redirect is still working), however this only happens on my country google results (google.com.pe), if you check my site on google.com, everything looks ok even with a sitelink, which I no longer have on my country search results. Back to the issue, the first thing I thought was go to Search Console and take it out from the index, so I asked for access by uploading a file but since everything on that old site redirects to my current site I can't make such action. While still waiting for such access, is there anything else I could do?. Thanks in advance. PD: I'm adding the images of my SERPs CmzN8kY G3zZwwj
Technical SEO | | JoaoCJ0 -
Choice of domain
Hi. I want to build a new site that is optimised for a training product that we have. We have an existing domain which I'm considering pointing at this new site. This domain is one of the new .training TLDs. Let's call this domain foo.training where my main keyword to optimise for will be "foo training". I've also looked and can see that foo-training.com is available. I read up on best practices for domains here : https://moz.com/learn/seo/domain My question is will the .training domain be seen as "spammy" in any way? Am I better to just go ahead and register the .com?
Technical SEO | | rmcatalyst0 -
Multiple sub domain appearing
Hi Everyone, Hope were well!. Have a strange one!!. New clients website http://www.allsee-tech.com. Just found out he is appearing for every subdomain possible. a.alsee-tech.com b.allsee-tech.com. I have requested htaccess as this is where I think the issue lies but he advises there isn't anything out of place there. Any ideas in case it isn't? Regards Neil
Technical SEO | | nezona0 -
Indexing Issue
Hi, I am working on www.stjohnswaydentalpractice.co.uk Google only seems to be indexing two of the pages when i search site:www.stjohnswaydentalpractice.co.uk I have added the site to webmaster tools and created a new sitemap which is showing that it has only submitted two of the pages. Can anyone shed any light for why these pages are not being indexed? Thanks Faye
Technical SEO | | dentaldesign0 -
More than one web domain
Hi What is best when considering using more than one domain on a website, what the best policy ? it's a question I get asked a lot, usually because prior to any seo efforts the main domain name purchased is not keyword rich or an abbreviated company name etc. what impact do new domains have on SEO compared to older existing domains- is it worth changing a generic company named domain for a keyword rich domain? if having multiple domains pointing to the site is beneficial how best is it to configure? How do i inform Google? How do both domains get index when there is only one physical site? How should i monitor it, Analytic s, SEO moz? Will the domain compete against each other and effect SEO rank? Are they best used for marketing purposes on external sites, adverts driving traffic to the main site. I'm aware there are lots of questions above, any answers/ opinions , links to further info would be greatly appreciated. cheers
Technical SEO | | Bristolweb0 -
301 Redirect Domain or 301 Redirect Domain + Interior Pages
Hello - My company acquired another company in our industry and our IT team immediately set up the acquired companies domain name as a an alias to our site. This created a duplicate version of our website under another domain name and Google started ranking interior pages from the aliased acquired site for several top keywords that were previously held by our real site. Should we 301 redirect just the top level domain name of the acquired site to the real site or 301 redirect the top level domain name and the interior pages on the acquired site to help ensure that our real domain will take back the rankings it once had? Thanks!
Technical SEO | | Room2140