Best way to block a sub-domain from being indexed
-
Hello,
The search engines have indexed a sub-domain I did not want indexed its on
old.domain.com and dev.domain.com - I was going to password them but is there a best practice way to block them.
My main domain default robots.txt says :-
Sitemap: http://www.domain.com/sitemap.xml
global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category//
Disallow: */trackback/
Disallow: */feed/
Disallow: /comments/
Disallow: /? -
Hi,
CleverPhD has some interesting ideas with robots.txt and Google Webmaster Tools, but simply password protecting all dev pages should keep pages out of Google's index. There's no best practice here, since a password wall will keep Googlebot out on its own.
To be doubly safe, you can also include a meta noindex tag on dev pages.
Keep in mind that once a page is in Google's index, it's going to take awhile for it to leave (unless you use CleverPhD's method). But, having a blank page in Google's index really isn't all that bad. It's there, but it won't rank for much.
Hope this helps,
Kristina
-
I've never tried a method like this - FreshFireOne, did you?
-
First and foremost when you finish all this - password protect your dev instances. A url will leak out eventually and then this happens. I know it is a PIA, but it is worth it.
To remove subdomains. Go into GWT and register the subdomains as separate websites in GWT. Create a robots.txt for each subdomain (not the one you mention, you need a robots that is specific to that subdomain that disallows all files. If you cant do that, have your subdomains include a noindex meta tag on all pages. You have to be careful with this as you do not want to push out your dev. robots.txt or the noindex meta tags to your production server, but it can be done. Talk to your devs. Then go into GWT and use the URL removal tool. Just leave it blank and it will remove the whole site.
Poof. Gone. You can then watch the GWT accounts. They will show errors for the dev site like "Severe health issues are found on your site - Some important page has been removed by request." This is a good error as it confirms that that subdomain is removed.
We actually used this not on a dev site but on our www1 server that was indexed. We use a load balancer with multiple copies of the site. www1 was completing with www. Using this above did the trick.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best way to structure website URLs ?
Hi, can anyone help me to understand if having category folder in URL matters or not? how to google treat a URL? for example, I have the URL www.protoexpress.com/pcb/certification but not sure google will treat it a whole or in separate parts? if in separate parts, is it safe to use pcb/pcb-certification? or it will be considered as keyword stuffing? Thank you in anticipation,
Intermediate & Advanced SEO | | SierraPCB1 -
If I block a URL via the robots.txt - how long will it take for Google to stop indexing that URL?
If I block a URL via the robots.txt - how long will it take for Google to stop indexing that URL?
Intermediate & Advanced SEO | | Gabriele_Layoutweb0 -
Best Approach to Redirect One Domain to Another
So I'm about to migrate one domain to another. Lets say I'm migrating boo.com to foo.com. Boo.com has good organic traffic & has some really well ranked pages. For this reason (I think) I want to send that traffic to some where other than the foo.com homepage. Perhaps a catered landing page. My question is can I redirect some of the specific pages on boo.com to a landing page on foo.com & then redirect the delta to foo.com's homepage? Or am a risking not fully transferring the full credit of one domain to another if I take that approach & therefore I should just redirect one domain to the other in its entirety? Thanks, Rich
Intermediate & Advanced SEO | | RPD0 -
301 many smaller domains to a new, large domain
Hi all, I have a question regarding permanently redirecting many small websites into one, large new one. During the past 9 years I have created many small websites, all focusing on hotel reservations in one specific city. This has served me beautifully in the past, but I have come to the conclusion that it is no longer a sustainable model and therefore I am in the process of creating one large, worldwide hotel reservations website. To not loose any benefit of my hard work the past 9 years, I want to permanently redirect the smaller websites to the correct section of my new website. I know that if it is only a few websites, that this strategy is perfectly acceptable, but since I am talking about 50 to 100 websites, I am not so sure and would like to have your input. Here is what I would like to do: (the domain names are not mine, just an example) Old website: londonhotels.com 301 to newdomain.com/london/ Old website: berlinhotels.com 301 to newdomain.com/berlin/ Old website: amsterdamhotels.com 301 to newdomain.com/amsterdam/ Etc., etc. My plan is to do this for 50 to 100 websites and would like to have your thoughts on if this is an acceptable strategy or not. Just to be clear, I am talking about redirecting only my websites that are in good standing, i.e. none of the websites I am thinking about 301'ing have been penalized. Thanks for your thoughts on this.
Intermediate & Advanced SEO | | tfbpa0 -
Aged domain and 301 redirect? (11 year old domain)
Hey everyone, I'm about to launch a new website for an accounting firm. They currently have a website, which has an 11 year old domain. They are doing very well locally for SEO, and i'm guessing it's because of the aged domain, as their website is very badly built, and contains almost no content. They would like to launch the new site with a simpler, easier to remember domain. If i launch the new site, point the aged domain using a 301 redirect, and do redirects for all of the old pages to the newer versions of them, is there a chance the company will lose their current SEO rankings? Thanks!
Intermediate & Advanced SEO | | RCDesign740 -
What is the best way to consolidate two websites into one?
Someone within our company's IT department just sent me some SEO advice that I believe is bogus. Can someone let me know if my initial gut-check is correct? We have two websites selling two identical catalogs of products but branded differently (color scheme, wording, etc.) like this: www.one.com
Intermediate & Advanced SEO | | Ryan-Ricketts
www.two.com We want to shut down the second website. I think we should set up 301 redirects from all pages on the second site to corresponding (relevant) pages on the first. In theory, this would pass over 90% of the earned link juice from one to the other. Here is what my IT peer said: "We could keep www.two.com set up indefinitely and just have it as the same web site as www.one.com (so two URLs but one site). This would help alleviate any issues with search engine results, etc. (Although I believe Ryan would agree this does impact www.one.com's rankings a bit, but shouldn't be a problem as long as we don't advertise both.) Google doesn't know they are on the same site, so you could technically get away with it. And it helps in indexing multiple pages on our sites." ... but wouldn't this be a big no-no because of the massive amounts of duplicate content it would create?0 -
Best Product URL For Indexing
My proposed URL: mydomain.com/products/category/subcategory/product detail Puts my products 4 levels deep. Is this too deep to get my products indexed?
Intermediate & Advanced SEO | | waynekolenchuk0 -
Domain certificate
Hello, I would like to know if there is any certificate we can buy to increase the Seo of my website. Thanks very much for your time
Intermediate & Advanced SEO | | Cartageno0