Block a sub-domain from being indexed
-
This is a pretty quick and simple (i'm hoping) question. What is the best way to completely block a sub domain from getting indexed from all search engines?
One item i cannot use is the meta "no follow" tag.
Thanks! - Kyle
-
Keep in mind that Google Index's everything that it can crawl. Even if you put a block in the robots.txt they will probably crawl it. You can require a password to that subdomain and keep big G out. This is easy to do if you have a site with cpanel access. Just go to manage permissions, and password protect that director with a .htaccess pw.
-
The robots.txt file just tells the bots you would "prefer" they don't index but there is nothing to prevent them from indexing.The only sure way to do this is to restrict access to the sub-domain for everyone and require some sort of authentication. If they don't have access they can't index.
-
In subdomain.example.com/robots.txt add the statements:
User-agent: *
Disallow: /Warning: Be absolutely certain that the above statements are not included in your example.com/robots.txt file or you'll kill your site.
-
Each subdomain may have its own robots.txt file. So for that subdomain, you can put:
User-agent: * Disallow: /In the robots.txt, and that should do it.
Please note that disallowing pages in robots.txt will not necessarily mean they won't appear on search result pages.... if people link to pages that are disallowed on that subdomain, they can still appear in SERPs. I had this happen with a few pages, which leads to funny listings in the SERPs because Google has to guess what the page title and description of the page should be, since it's not allowed to read the page. The meta noindex tag is the way to go if you want to be really sure the page doesn't appear in the SERPs. If you use that, don't disallow the page. Here's a recent SEOMoz post about it: http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts
-
That was going to be my assumption but i wasn't 100% sure how they worked with sub domains. Are you able to supply a little more information on implementation? It is extremely important that it only blocks: sub.domain.com and not domain.com
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google is indexing our old domain
We changed our primary domain from vivitecsolutions.com to vivitec.net. Google is indexing our new domain, but still has our old domain indexed too. The problem is that the old site is timing out because of the https: Thought on how to make the old indexing go away or properly forward the https?
Technical SEO | | AdsposureDev0 -
Recently re-built our site and changed domain. Now I want to go back to old domain - it it a bad idea?
About a year ago I rebuilt our website and changed our domain name. We rent villas in Tuscany, we used to be 'invitationtotuscany.com'. Then I started doing the same in Provence, and in the italian lakes, so i had further sites called invitationtoprovence.com and invitationtotheitalianlakes.com. But maintaining them was awkward and I wanted to have one site. So I put them all onto invitationto.com and did 301s from the old domains and sites. Now I'd dropped off organic search results and I've also realised that invitationto.com is far less clear as a business address. My inclination is to go back to invitationtotuscany.com - Tuscany is still 80% of our business and have the other areas in there too - optimised for SEO for Provence etc. I'm being told its a really bad idea to change domain, 301 the old one, and then revert to the original domain. But I'm suffering anyway, so I wonder if I sjhouldn't just bite the bullet. A lot of my old good backlinks still point to invitationtotuscany.com (BBC, Sunday Times, etc) and the DA is 33 against 22 on the new one.. All help gratefully received! : )
Technical SEO | | DanWrightson0 -
Spammers created bad links to old hacked domain, now redirected to our new domain. Advice?
My client had an old site hacked (let's call it "myolddomain.com") and the hackers created many links in other hacked sites with links such as http://myolddomain.com/styless.asp?jordan-12-taxi-kids-cheap-T8927.html The old myolddomain.com site was redirected to a different new site since then, but we still see over a thousand spam links showing up in the new site's Search Console 404 crawl errors report. Also, using the links: operator in google search, we see many results of spam links. Should we be worried about these bad links pointing to our old site and redirecting to 404s on the new site? What is the best recommendation to clean them up? Ignore? 410s? Other? I'm seeing conflicting advice out there. The old site is hosted by the client's previous web developer who doesn't want to clean anything up on their end without an ongoing hosting contract. So beyond turning redirects on or off, the client doesn't want to pay for any additional hosting. So we don't have much control over anything related to "myolddomain.com". 😞 Thanks in advance for any assistance!
Technical SEO | | usDragons0 -
Keyword domain usage
Hi I have a client who wants some advice on the following and I haven't come up with anything decent yet. So wanted to throw it out here. They own a domain like soccershoes.com which also houses their web site. They have now bought ussoccershoes.com because this US based search term is more popular than simply soccer shoes. How can they utilise this new domain effectively to drive traffic to their main web site? Thanks for any input.
Technical SEO | | yolkcreative0 -
More than one web domain
Hi What is best when considering using more than one domain on a website, what the best policy ? it's a question I get asked a lot, usually because prior to any seo efforts the main domain name purchased is not keyword rich or an abbreviated company name etc. what impact do new domains have on SEO compared to older existing domains- is it worth changing a generic company named domain for a keyword rich domain? if having multiple domains pointing to the site is beneficial how best is it to configure? How do i inform Google? How do both domains get index when there is only one physical site? How should i monitor it, Analytic s, SEO moz? Will the domain compete against each other and effect SEO rank? Are they best used for marketing purposes on external sites, adverts driving traffic to the main site. I'm aware there are lots of questions above, any answers/ opinions , links to further info would be greatly appreciated. cheers
Technical SEO | | Bristolweb0 -
Multiple domains on the same hosting
Hello, I have around 80 sites (different domains) which are on the same host. Everytime I look up the IP of these domains, it gives me the same IP. The sites are interlinked, all linking to the main domain. My question is: Does the off site link building have effect on the 80 sites?
Technical SEO | | Arianittt0 -
Two blogs on the same domain
I have had two blogs on the same domain for a while now, and it just occurred to me that no one else seems to do this and maybe it's even weird. http://www.stadriemblems.com/blog/
Technical SEO | | UnderRugSwept
http://www.stadriemblems.com/scouting/blog/ One is our main blog, and one is for a very concentrated niche of customers. What are your opinions on this? Everything from SEO to best practices, to overall unusual-ness?0 -
Checkout on different domain
Is it a bad SEO move to have a your checkout process on a separate domain instead of the main domain for a ecommerce site. There is no real content on the checkout pages and they are completely new pages that are not indexed in the search engines. Do to the backend architecture it is impossibe for us to have them on the same domain. An example is this page: http://www.printingforless.com/2/Brochure-Printing.html One option we've discussed to not pass page rank on to the checkout domain by iFraming all of the links to the checkout domain. We could also move the checkout process to a subdomain instead of a new domain. Please ignore the concerns with visitors security and conversion rate. Thanks!
Technical SEO | | PrintingForLess.com0