Block a sub-domain from being indexed
-
This is a pretty quick and simple (i'm hoping) question. What is the best way to completely block a sub domain from getting indexed from all search engines?
One item i cannot use is the meta "no follow" tag.
Thanks! - Kyle
-
Keep in mind that Google Index's everything that it can crawl. Even if you put a block in the robots.txt they will probably crawl it. You can require a password to that subdomain and keep big G out. This is easy to do if you have a site with cpanel access. Just go to manage permissions, and password protect that director with a .htaccess pw.
-
The robots.txt file just tells the bots you would "prefer" they don't index but there is nothing to prevent them from indexing.The only sure way to do this is to restrict access to the sub-domain for everyone and require some sort of authentication. If they don't have access they can't index.
-
In subdomain.example.com/robots.txt add the statements:
User-agent: *
Disallow: /Warning: Be absolutely certain that the above statements are not included in your example.com/robots.txt file or you'll kill your site.
-
Each subdomain may have its own robots.txt file. So for that subdomain, you can put:
User-agent: * Disallow: /In the robots.txt, and that should do it.
Please note that disallowing pages in robots.txt will not necessarily mean they won't appear on search result pages.... if people link to pages that are disallowed on that subdomain, they can still appear in SERPs. I had this happen with a few pages, which leads to funny listings in the SERPs because Google has to guess what the page title and description of the page should be, since it's not allowed to read the page. The meta noindex tag is the way to go if you want to be really sure the page doesn't appear in the SERPs. If you use that, don't disallow the page. Here's a recent SEOMoz post about it: http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts
-
That was going to be my assumption but i wasn't 100% sure how they worked with sub domains. Are you able to supply a little more information on implementation? It is extremely important that it only blocks: sub.domain.com and not domain.com
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Follow no-index
I have a question about the right way to not index pages: With a canonical or follow no-index. First we have a blog page: **Blogpage **
Technical SEO | | Happy-SEO
URL: /blog/
index follow Page 2 blog:
URL: /blog?=p2
index follow
rel="prev" /blog/
el="next" ?=p3 Nothing strange here i guess. But we also have other pages with chance on duplicate content: /SEO-category/
/SEO-category/view-more/ Because i don't want the "view-more" items to be indexed i want to set it on: follow no-index (follow to reach pages). But now the "view-more" also have pagination. What is the best way? Option 1:
/SEO-category/view-more/
Follow no-index /SEO-category/view-more?=p2
Follow no-index
rel="prev" /view-more/
el="next" ?=p3 Option 2: /SEO-category/view-more/
Canonical: /SEO-category/ /SEO-category/view-more?=p2
rel="prev" /view-more/
el="next" ?=p3 Option 3: Other suggests? Thanks!0 -
Is repurposing an old sub domain better than creating a new sub domain?
We have a good sub domain like** art.ourwebsite.com** which currently sells custom canvas art. We have owned the domain since 2013 but it has only been live for the past few weeks. We want to redesign & repurpose the page to continue to sell custom canvas art but will eventually include other merchandise like mugs, tshirts, etc which wouldn't be custom. Would it be best to keep art.ourwebsite.com since is a shorter/more memorible & older sub domain or would it be best to update the name to something that encompasses our new products? Our marketing team has suggested yourart.ourwebsite.com
Technical SEO | | sb10301 -
How to redirect old domain to new domain.
We just recently signed up to Moz with hopes of fixing our Moz Ranking. We have an old domain - http://at-net.net and a new domain - https://www.expertip.net We have set up 301 (Permanent) redirects from all pages on the old site to the new, but aren't getting the ranking or aren't getting recognized from external links to the old sites. I've read the moz article on 'Link Juice' and followed those practices, but it doesn't seem to help. Does anyone have advice on doing this? Thanks in advance,
Technical SEO | | greg.lanier
Greg0 -
Images on sub domain fed from CDN
I have a client that uses a CDN to fill images, from a sub domain ( images.domain.com). We've made sure that the sub domain itself is not blocked. We've added a robots.txt file, we're creating an image sitemap file & we've verified ownership of the domain within GWT. Yet, any crawler that I use only see's the first page of the sub domain (which is .html) but none of the subsequent URL's which are all .jpeg. Is there something simple I'm missing here?
Technical SEO | | TammyWood0 -
What is the advantage of using sub domains instead of pages on the root domain?
Have a look at this example http://bannerad.designcrowd.com/ For each category of design, they have a landing page on the sub domain. Wouldn't it be better to have them as part of the same domain? What is the strategy behind using sub domains?
Technical SEO | | designquotes0 -
Homepage/Root domain de-indexed by Google
This morning I discovered that the homepage/root domain of our company site, http://www.collegeplus.org/, has been de-indexed by Google and Bing. Out IT dept. is claiming it's our fault because we changed the meta title on our homepage. But they will not give me access to GWT to see if there's any issues. I believe the issue lies within our robots.txt file - http://www.collegeplus.org/robots.txt I also don't believe we're suffering a penalty because all of our tier 2 pages are still indexed when any type of branded search is performed. We don't do things that can get a site de-indexed like this. Any ideas on what the issue may be? Or at least something to convince our IT dept. that simply changing a meta title won't get your homepage totally de-indexed? Thanks.
Technical SEO | | explorionary0 -
Switch domain from one account to another at domain registrar
We need to move our domain from one account to another at our domain registrar (which is Moniker). Both the "from" account and "to" account will be at Moniker. The "from" account currently has privacy settings enabled and we'd also put these in place for the "to" account. Has anyone done this and see any impact on SEO? Is there any big or common mistakes that I should be aware of? Thanks all!
Technical SEO | | evoNick0 -
Sub-Domain Choice Dilemma
We have successfully rolled out 5 sub-domains using very industry specific KWs as the sub, e.g. familylawyers. We're rolling our an employment focused sub.and ideally would use employmentlawyers.XXX.com. However I'm tempted to use a long established (5+ years) sub-domain with a topic related KW that now hosts a non-active blog with PR3 - employment-law.XXXX.com. It has 49 indexed pages, some with PR2. So there's potential for getting a kickstart on traffic and trust with some redirects. Should I go for instant gratification or build for the long haul with the slightly more beneficial URL KWing? I should add that this sub-domain will have thousands of pages that are geo and sub-category focused - a typical URL would be. sub-domain.XXX.com/TX/Austin/wrongful-termination-lawyers.html THX for any opinions.
Technical SEO | | legalseo0