Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      Enhance Keyword Discovery with Bulk Analysis
      Moz Pro

      Enhance Keyword Discovery with Bulk Analysis

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Access 20 years of data with flexible pricing
      Moz API

      Access 20 years of data with flexible pricing

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Intermediate & Advanced SEO
    4. Best way to block a sub-domain from being indexed

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Best way to block a sub-domain from being indexed

    Intermediate & Advanced SEO
    3
    4
    2531
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • JohnW-UK
      JohnW-UK last edited by

      Hello,

      The search engines have indexed a sub-domain I did not want indexed its on

      old.domain.com and dev.domain.com - I was going to password them but is there a best practice way to block them.

      My main domain default robots.txt says :-

      Sitemap: http://www.domain.com/sitemap.xml

      global

      User-agent: *
      Disallow: /cgi-bin/
      Disallow: /wp-admin/
      Disallow: /wp-includes/
      Disallow: /wp-content/plugins/
      Disallow: /wp-content/cache/
      Disallow: /wp-content/themes/
      Disallow: /trackback/
      Disallow: /feed/
      Disallow: /comments/
      Disallow: /category//
      Disallow: */trackback/
      Disallow: */feed/
      Disallow: /comments/
      Disallow: /
      ?

      1 Reply Last reply Reply Quote 0
      • KristinaKledzik
        KristinaKledzik last edited by

        Hi,

        CleverPhD has some interesting ideas with robots.txt and Google Webmaster Tools, but simply password protecting all dev pages should keep pages out of Google's index. There's no best practice here, since a password wall will keep Googlebot out on its own.

        To be doubly safe, you can also include a meta noindex tag on dev pages.

        Keep in mind that once a page is in Google's index, it's going to take awhile for it to leave (unless you use CleverPhD's method). But, having a blank page in Google's index really isn't all that bad. It's there, but it won't rank for much.

        Hope this helps,

        Kristina

        1 Reply Last reply Reply Quote 0
        • KristinaKledzik
          KristinaKledzik @CleverPhD last edited by

          I've never tried a method like this - FreshFireOne, did you?

          1 Reply Last reply Reply Quote 0
          • CleverPhD
            CleverPhD last edited by

            First and foremost when you finish all this - password protect your dev instances.  A url will leak out eventually and then this happens.  I know it is a PIA, but it is worth it.

            To remove subdomains.  Go into GWT and register the subdomains as separate websites in GWT.  Create a robots.txt for each subdomain (not the one you mention, you need a robots that is specific to that subdomain that disallows all files.  If you cant do that, have your subdomains include a noindex meta tag on all pages.   You have to be careful with this as you do not want to push out your dev. robots.txt or the noindex meta tags to your production server, but it can be done.  Talk to your devs.    Then go into GWT and use the URL removal tool.  Just leave it blank and it will remove the whole site.

            Poof.  Gone.  You can then watch the GWT accounts.  They will show errors for the dev site like "Severe health issues are found on your site - Some important page has been removed by request."  This is a good error as it confirms that that subdomain is removed.

            We actually used this not on a dev site but on our www1 server that was indexed.  We use a load balancer with multiple copies of the site.  www1 was completing with www.   Using this above did the trick.

            KristinaKledzik 1 Reply Last reply Reply Quote 1
            • 1 / 1
            • First post
              Last post

            Got a burning SEO question?

            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


            Start my free trial


            Browse Questions

            Explore more categories

            • Moz Tools

              Chat with the community about the Moz tools.

            • SEO Tactics

              Discuss the SEO process with fellow marketers

            • Community

              Discuss industry events, jobs, and news!

            • Digital Marketing

              Chat about tactics outside of SEO

            • Research & Trends

              Dive into research and trends in the search industry.

            • Support

              Connect on product support and feature requests.

            • See all categories

            Related Questions

            • gozmoz

              Moving html site to wordpress and 301 redirect from index.htm to index.php or just www.example.com

              I found page duplicate content when using Moz crawl tool, see below. http://www.example.com
              Page Authority 40
              Linking Root Domains 31
              External Link Count 138
              Internal Link Count 18
              Status Code 200
              1 duplicate http://www.example.com/index.htm
              Page Authority 19
              Linking Root Domains 1
              External Link Count 0
              Internal Link Count 15
              Status Code 200
              1 duplicate I have recently transfered my old html site to wordpress.
              To keep the urls the same I am using a plugin which appends .htm at the end of each page. My old site home page was index.htm. I have created index.htm in wordpress as well but now there is a conflict of duplicate content. I am using latest post as my home page which is index.php Question 1.
              Should I also use redirect 301 im htaccess file to transfer index.htm page authority (19) to www.example.com If yes, do I use
              Redirect 301 /index.htm http://www.example.com/index.php
              or
              Redirect 301 /index.htm http://www.example.com Question 2
              Should I change my "Home" menu link to http://www.example.com instead of http://www.example.com/index.htm that would fix the duplicate content, as indx.htm does not exist anymore. Is there a better option? Thanks

              Intermediate & Advanced SEO | | gozmoz
              0
            • atomiconline

              Best way to "Prune" bad content from large sites?

              I am in process of pruning my sites for low quality/thin content. The issue is that I have multiple sites with 40k + pages and need a more efficient way of finding the low quality content than looking at each page individually. Is there an ideal way to find the pages that are worth no indexing that will speed up the process but not potentially harm any valuable pages? Current plan of action is to pull data from analytics and if the url hasn't brought any traffic in the last 12 months then it is safe to assume it is a page that is not beneficial to the site. My concern is that some of these pages might have links pointing to them and I want to make sure we don't lose that link juice. But, assuming we just no index the pages we should still have the authority pass along...and in theory, the pages that haven't brought any traffic to the site in a year probably don't have much authority to begin with. Recommendations on best way to prune content on sites with hundreds of thousands of pages efficiently?  Also, is there a benefit to no indexing the pages vs deleting them? What is the preferred method, and why?

              Intermediate & Advanced SEO | | atomiconline
              0
            • clarkovitch

              Legacy domains

              Hi all, A couple of years ago we amalgamated five separate domains into one, and set up 301 redirects from all the pages on the old domains to their equivalent pages on the new site. We were a bit tardy in using the "change of address" tool in Search Console, but that was done nearly 8 months ago now as well. Two years after implementing all the redirects, the old domains still have significant authority (DAs of between 20-35) and some strong inbound links. I expected to see the DA of the legacy domains taper off during this period and (hopefully!) the DA of the new domain increase. The latter has happened, although not as much as I'd hoped, but the DA of the legacy domains is more or less as good as it ever was? Google is still indexing a handful of links from the legacy sites, strangely even when it is picking up the redirects correctly. So, for example, if you do a site:legacydomain1.com query, it will give a list of results which includes pages where it shows the title and snippet of the page on newdomain.com, but the link is to the page on legacydomain1.com. What has prompted me to finally try and resolve this is that the server which hosted the original 5 domains is now due to be decommissioned which obviously means the 301 redirects for the original pages will no longer be served. I can set up web forwarding for each of the legacy domains at the hosting level, but to maintain the page-by-page redirects I'd have to actually host the websites somewhere. I'd like to know the best way forward both in terms of the redirect issue, and also in terms of the indexing of the legacy domains? Many thanks, Dan

              Intermediate & Advanced SEO | | clarkovitch
              0
            • dbaxa-261338

              Consolidating Multiple Domains into A Single Domain

              I have a client who's website is an amalgamation of multiple domains. jacksonhole.net is the main domain but the site passes traffic back and forth from the following domains/sites. My questions is, would it it be better for SEO to consolidate all of these domains under the single high authority domain and 301 redirect the rest or is that a really bad idea?  Thanks for your help. jacksonhole.net (Domain Authority 31) jackson-hole-rental-condos.com (Domain Authority 22) jackson-hole-rental-homes.com (Domain Authority 21) j acksonholehotelguide.com (Domain Authority 19)

              Intermediate & Advanced SEO | | dbaxa-261338
              1
            • McTaggart

              Sitemaps during a migration - which is the best way of dealing with them?

              Many SEOs I know simply upload the new sitemap once the new site is launched - some keep the old site's URLs on the new sitemap (for a while) to facilitate the migration - others upload both the old and the new website together, to support the migration. Which is the best way to proceed? Thanks, Luke

              Intermediate & Advanced SEO | | McTaggart
              0
            • steviephil

              Best way to noindex an image?

              Hi all, A client wanted a few pages noindexed, which was no problem using the meta robots noindex tag. However they now want associated images removed, some of which still appear on pages that they still want indexed. I added the images to their robots.txt file a few weeks ago (probably over a month ago actually) but they're all still showing when you do an image search. What's the best way to noindex them for good, and how do I go about implementing it? Many thanks, Steve

              Intermediate & Advanced SEO | | steviephil
              0
            • JohnW-UK

              One Way Links vs Two Way Links

              Hi, Was speaking to a client today and got asked how damaging two way links are. i.e. domaina.com links to domainb.com and domainb.com links back to domaina.com. I need a nice simple layman's explanation of if/how damaging they are compared to one way links. And please don't answer with you lose link juice as I have a job explaining link juice.... I am explaining things to a non techie! Thank you!!

              Intermediate & Advanced SEO | | JohnW-UK
              0
            • SeoSheikh

              Should I buy a .co domain if my preferred .com and .co.uk domain are taken by other companies?

              I'm looking to boost my website ranking and drive more traffic to it using a keyword rich domain name. I want to have my nearest city followed by the keyword "seo" in the domain name but the .co.uk and .com have already been taken. Should I take the plunge and buy .co at a higher price? What options do I have? Also whilst we're on domains and URL's is it best to separate keywords in url's with a (_) or a (-)? Many thanks for any help with this matter. Alex

              Intermediate & Advanced SEO | | SeoSheikh
              0

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy

            Looks like your connection to Moz was lost, please wait while we try to reconnect.