Block a sub-domain from being indexed
-
This is a pretty quick and simple (i'm hoping) question. What is the best way to completely block a sub domain from getting indexed from all search engines?
One item i cannot use is the meta "no follow" tag.
Thanks! - Kyle
-
Keep in mind that Google Index's everything that it can crawl. Even if you put a block in the robots.txt they will probably crawl it. You can require a password to that subdomain and keep big G out. This is easy to do if you have a site with cpanel access. Just go to manage permissions, and password protect that director with a .htaccess pw.
-
The robots.txt file just tells the bots you would "prefer" they don't index but there is nothing to prevent them from indexing.The only sure way to do this is to restrict access to the sub-domain for everyone and require some sort of authentication. If they don't have access they can't index.
-
In subdomain.example.com/robots.txt add the statements:
User-agent: *
Disallow: /Warning: Be absolutely certain that the above statements are not included in your example.com/robots.txt file or you'll kill your site.
-
Each subdomain may have its own robots.txt file. So for that subdomain, you can put:
User-agent: * Disallow: /In the robots.txt, and that should do it.
Please note that disallowing pages in robots.txt will not necessarily mean they won't appear on search result pages.... if people link to pages that are disallowed on that subdomain, they can still appear in SERPs. I had this happen with a few pages, which leads to funny listings in the SERPs because Google has to guess what the page title and description of the page should be, since it's not allowed to read the page. The meta noindex tag is the way to go if you want to be really sure the page doesn't appear in the SERPs. If you use that, don't disallow the page. Here's a recent SEOMoz post about it: http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts
-
That was going to be my assumption but i wasn't 100% sure how they worked with sub domains. Are you able to supply a little more information on implementation? It is extremely important that it only blocks: sub.domain.com and not domain.com
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We switched the domain from www.blog.domain.com to domain.com/blog.
We switched the domain from www.blog.domain.com to domain.com/blog. This was done with the purpose of gaining backlinks to our main website as well along with to our blog. This set us very low in organic traffic and not to mention, lost the backlinks. For anything, they are being redirected to 301 code. Kindly suggest changes to bring back all the traffic.
Technical SEO | | arun.negi0 -
Domain change recommendations
We recently migrated one of our websites to a new domain. Obviously we were expecting a decrease in traffic initially, but it has actually gone down by 70% week-over-week since we made the switch. We set up a 301 redirect from the old domain to the new domain, changed all internal links to the new domain and changed all inbound links that we owned to the new domain. Our research suggested the best way to approach a domain change was by keeping it simple and not making too many changes at once. So my questions are: 1. Are these the kinds of results we should expect initially after a domain change? And if not, 2. What are the steps we should take from here? Thanks!
Technical SEO | | gouldtr0 -
Migrating domains from a domain that will have new content.
We have a new url. The old url is being taken over by someone else. Is it possible to still have a successful redirect/migration strategy if we are redirect from our old domain, which is now being used by someone else. I see a big mess, but I'm being told we can redirect all the links to our old content (which is now used by someone else) to our new url. Thoughts? craziness? insanity? Or I'm just not getting it:)
Technical SEO | | CC_Dallas0 -
Two blogs on the same domain
I have had two blogs on the same domain for a while now, and it just occurred to me that no one else seems to do this and maybe it's even weird. http://www.stadriemblems.com/blog/
Technical SEO | | UnderRugSwept
http://www.stadriemblems.com/scouting/blog/ One is our main blog, and one is for a very concentrated niche of customers. What are your opinions on this? Everything from SEO to best practices, to overall unusual-ness?0 -
.com or .co.uk in UK index? but the .com has higher domain authority...
Hi there i have a .com and a .co.uk for a site that has been around a while. However not much seo has been done on it, i was wonderign do i continue to optimise for the .com or switch to the .co.uk to rank in Google UK index for various search terms. .COM = 40 domain authority .CO.UK - 10 domain authority. Let the debate start 🙂
Technical SEO | | pauledwards0 -
Switch domain from one account to another at domain registrar
We need to move our domain from one account to another at our domain registrar (which is Moniker). Both the "from" account and "to" account will be at Moniker. The "from" account currently has privacy settings enabled and we'd also put these in place for the "to" account. Has anyone done this and see any impact on SEO? Is there any big or common mistakes that I should be aware of? Thanks all!
Technical SEO | | evoNick0 -
Redirecting root domains to sub domains
Mozzers: We have a instance where a client is looking to 301 a www.example.com to www.example.com/shop I know of several issues with this but wondered if anyone could chip in with any previous experiences of doing so, and what outcomes positive and negative came out of this. Issues I'm aware of: The root domain URL is the most linked page, a HTTP 301 redirect only passes about 90% of the value. you'll loose 10-15% of your link value of these links. navigational queries (i.e.: the "domain part" of "domain.tld") are less likely to produce google site-links less deep-crawling: google crawls top down - starts with the most linked page, which will most likely be your domain url. as this does not exist you waste this zero level of crawling depth. robots.txt is only allowed on the root of the domain. Your help as always is greatly appreciated. Sean
Technical SEO | | Yozzer0 -
Exact match domains
I know buying new websites hoping to 301 redirect them to achieve higher rankings before they’re established is not a good idea, but what about uploading a info pages + on-page SEO + some link-building and when the site is established (it ranks) direct visitors to a landing page? Buying a new domain for the SINGLE purpose of 301 redirecting won’t boost the rankings (don't think it would get penalties either, unless it’s the only link building activity). Not planning to redirect to the homepage, but to the related sub page on our main site (specific landing page). Will this pass the appropriate anchor text and link authority to the right page on our website and help those pages to rank for their keywords? Only thinking of a few pages (no more than 3).
Technical SEO | | Diana.varbanescu0