Best way to block a sub-domain from being indexed
-
Hello,
The search engines have indexed a sub-domain I did not want indexed its on
old.domain.com and dev.domain.com - I was going to password them but is there a best practice way to block them.
My main domain default robots.txt says :-
Sitemap: http://www.domain.com/sitemap.xml
global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category//
Disallow: */trackback/
Disallow: */feed/
Disallow: /comments/
Disallow: /? -
Hi,
CleverPhD has some interesting ideas with robots.txt and Google Webmaster Tools, but simply password protecting all dev pages should keep pages out of Google's index. There's no best practice here, since a password wall will keep Googlebot out on its own.
To be doubly safe, you can also include a meta noindex tag on dev pages.
Keep in mind that once a page is in Google's index, it's going to take awhile for it to leave (unless you use CleverPhD's method). But, having a blank page in Google's index really isn't all that bad. It's there, but it won't rank for much.
Hope this helps,
Kristina
-
I've never tried a method like this - FreshFireOne, did you?
-
First and foremost when you finish all this - password protect your dev instances. A url will leak out eventually and then this happens. I know it is a PIA, but it is worth it.
To remove subdomains. Go into GWT and register the subdomains as separate websites in GWT. Create a robots.txt for each subdomain (not the one you mention, you need a robots that is specific to that subdomain that disallows all files. If you cant do that, have your subdomains include a noindex meta tag on all pages. You have to be careful with this as you do not want to push out your dev. robots.txt or the noindex meta tags to your production server, but it can be done. Talk to your devs. Then go into GWT and use the URL removal tool. Just leave it blank and it will remove the whole site.
Poof. Gone. You can then watch the GWT accounts. They will show errors for the dev site like "Severe health issues are found on your site - Some important page has been removed by request." This is a good error as it confirms that that subdomain is removed.
We actually used this not on a dev site but on our www1 server that was indexed. We use a load balancer with multiple copies of the site. www1 was completing with www. Using this above did the trick.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best SEO way to categorize products on an ecommerce site
What is the best way for SEO to set up categories for an ecommerce site selling beauty products. I have currently built my product categories so that if a person looks under the hydration category they find our body lotion, but also if they look under the body section of products they also will find the same body lotion. Is this a problem for SEO? I think it helps the customer find the product.
Intermediate & Advanced SEO | | Kuhliff0 -
Sub-domain or not???
Hi, We're setting up a forum for our users (our target audience responds extremely well to forums). I was wondering if it should be set up on a sub-domain or not. I'm leaning towards sub-domain, but our devs say this will impact how they approach it so I'd like to give them an answer asap so we can proceed with planning it! Thanks, Amelia
Intermediate & Advanced SEO | | CommT0 -
Is there a way to increase domain authority?
Dear all, when I see moz analytics for my blog irctcloginindia.co.in, it is legging behind only in terms of diomain authority when compared to my competitors. Because of which it is ranking low. Is there any short cut or fast method using which I can increase the authority for my domain.
Intermediate & Advanced SEO | | irctclogin0 -
Best Practices
Okay this would be a piece of cake for most of you out there.. What are the best practices once you add a page or piece of content on your website with a new keyword that you have never used before but plan to use it with every relevant new page you add. How do you ensure that Google will crawl that page? Secondly, if you add the new keyword in the old pieces of content/ pages you have already published by editing the content to suit that keyword, how would you ensure that it gets crawled my Google. Thanks in advance
Intermediate & Advanced SEO | | LaythDajani0 -
Consolidating two separate domains and redirecting towards a new replatformed domain
A client has two different sites selling the same products with the same content, they would like to replatform onto Magento while redirecting those 2 sites to the new URL. The question is, besides monitoring the 301 redirects is there anything else to take into consideration when consolidating two sites into one new site?
Intermediate & Advanced SEO | | RocketWeb0 -
Should I redirect my Google Update Effected Domain to brand new Domain?
Hey Moz experts, I had a domain which was really doing better but after the Humming Bird update my traffic was decreased up to 90%. There are plenty of posts on my existing blog, Now what should I do? I mean should I redirect it to a brand new domain or Copy all the posts to a brand new domain and delete my existing domain? Note that the Old domain has PR1, DA 19 and PA 30.
Intermediate & Advanced SEO | | imran20780 -
Is it safe to 301 redirect old domain to new domain after a manual unnatural links penalty?
I have recently taken on a client that has been manually penalised for spammy link building by two previous SEOs. Having just read this excellent discussion, http://www.seomoz.org/blog/lifting-a-manual-penalty-given-by-google-personal-experience I am weighing up the odds of whether it's better to cut losses and recommend moving domains. I had thought under these circumstances it was important not to 301 the old domain to the new domain but the author (Lewis Sellers) comments on 3/4/13 that he is aware of forwards having been implemented without transferring the penalty to the new domain. http://www.seomoz.org/blog/lifting-a-manual-penalty-given-by-google-personal-experience#jtc216689 Is it safe to 301? What's the latest thinking?
Intermediate & Advanced SEO | | Ewan.Kennedy0 -
Issues with Sub domains for dealers
I'm starting a new SEO project and am feeling a little overwhelmed due to the scale of it. I am not sure where to start and hope that someone has some ideas. Thousands of dealer websites reside as sub domains on gravelymower.com/ (e.g. http://quality-mowers.gravelymower.com/) The particular sub domain mentioned above is not showing up at all for any searches and is not cached by Google: http://webcache.googleusercontent.com/search?q=cache:http://quality-mowers.gravelymower.com/ I realize that pretty much zero SEO best practices are followed on page and the location is not on the page, but why is this sub domain not even being indexed by Google? Any help is appreciated. Thanks!
Intermediate & Advanced SEO | | BridgelineDigital880