Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      What is your Brand Authority?
      Moz

      What is your Brand Authority?

      Check yours now
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Intermediate & Advanced SEO
    4. Robots.txt: how to exclude sub-directories correctly?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Robots.txt: how to exclude sub-directories correctly?

    Intermediate & Advanced SEO
    3
    10
    53113
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • fablau
      fablau last edited by

      Hello here,

      I am trying to figure out the correct way to tell SEs to crawls this:

      http://www.mysite.com/directory/

      But not this:

      http://www.mysite.com/directory/sub-directory/

      or this:

      http://www.mysite.com/directory/sub-directory2/sub-directory/...

      But with the fact I have thousands of sub-directories with almost infinite combinations, I can't put the following definitions in a manageable way:

      disallow: /directory/sub-directory/

      disallow: /directory/sub-directory2/

      disallow: /directory/sub-directory/sub-directory/

      disallow: /directory/sub-directory2/subdirectory/

      etc...

      I would end up having thousands of definitions to disallow all the possible sub-directory combinations.

      So, is the following way a correct, better and shorter way to define what I want above:

      allow: /directory/$

      disallow: /directory/*

      Would the above work?

      Any thoughts are very welcome! Thank you in advance.

      Best,

      Fab.

      1 Reply Last reply Reply Quote 1
      • MickEdwards
        MickEdwards @sjunaidali last edited by

        I mentioned both.  You add a meta robots to noindex and remove from the sitemap.

        1 Reply Last reply Reply Quote 0
        • sjunaidali
          sjunaidali @MickEdwards last edited by

          But google is still free to index a link/page even if it is not included in xml sitemap.

          MickEdwards 1 Reply Last reply Reply Quote 0
          • MickEdwards
            MickEdwards @sjunaidali last edited by

            Install Yoast Wordpress SEO plugin and use that to restrict what is indexed and what is allowed in a sitemap.

            sjunaidali 1 Reply Last reply Reply Quote 1
            • sjunaidali
              sjunaidali @MickEdwards last edited by

              I am using wordpress, Enfold theme (themeforest).

              I want some files to be accessed by google, but those should not be indexed.

              Here is an example: http://prntscr.com/h8918o

              I have currently blocked some JS directories/files using robots.txt (check screenshot)

              But due to this I am not able to pass Mobile Friendly Test on Google: http://prntscr.com/h8925z (check screenshot)

              Is its possible to allow access, but use a tag like noindex in the robots.txt file. Or is there any other way out.

              MickEdwards 1 Reply Last reply Reply Quote 0
              • fablau
                fablau last edited by

                Yes, everything looks good, Webmaster Tools gave me the expected results with the following directives:

                allow: /directory/$

                disallow: /directory/*

                Which allows this URL:

                http://www.mysite.com/directory/

                But doesn't allow the following one:

                http://www.mysite.com/directory/sub-directory2/...

                This page also gives an update similar to mine:

                https://support.google.com/webmasters/answer/156449?hl=en

                I think I am good! Thanks 🙂

                1 Reply Last reply Reply Quote 2
                • fablau
                  fablau last edited by

                  Thank you Michael, it is my understanding then that my idea of doing this:

                  allow: /directory/$

                  disallow: /directory/*

                  Should work just fine. I will test it within Google Webmaster Tools, and let you know if any problems arise.

                  In the meantime if anyone else has more ideas about all this and can confirm me that would be great!

                  Thank you again.

                  1 Reply Last reply Reply Quote 1
                  • MickEdwards
                    MickEdwards @fablau last edited by

                    I've always stuck to Disallow and followed -

                    "This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory:"

                    http://www.robotstxt.org/robotstxt.html

                    From https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt this seems contradictory

                    | /* | equivalent to / | equivalent to / | Equivalent to "/" -- the trailing wildcard is ignored. |

                    I think this post will be very useful  for you - http://moz.com/community/q/allow-or-disallow-first-in-robots-txt

                    1 Reply Last reply Reply Quote 1
                    • fablau
                      fablau @MickEdwards last edited by

                      Thank you Michael,

                      Google and other SEs actually recognize the "allow:" command:

                      https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

                      The fact is: if I don't specify that, how can I be sure that the following single command:

                      disallow: /directory/*

                      Doesn't prevent SEs to spider the /directory/ index as I'd like to?

                      MickEdwards 1 Reply Last reply Reply Quote 0
                      • MickEdwards
                        MickEdwards last edited by

                        As long as you dont have directories somewhere in /* that you want indexed then I think that will work.  There is no allow so you don't need the first line just

                        disallow: /directory/*

                        You can test out here- https://support.google.com/webmasters/answer/156449?rd=1

                        fablau sjunaidali 2 Replies Last reply Reply Quote 0
                        • 1 / 1
                        • First post
                          Last post

                        Got a burning SEO question?

                        Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                        Start my free trial


                        Browse Questions

                        Explore more categories

                        • Moz Tools

                          Chat with the community about the Moz tools.

                        • SEO Tactics

                          Discuss the SEO process with fellow marketers

                        • Community

                          Discuss industry events, jobs, and news!

                        • Digital Marketing

                          Chat about tactics outside of SEO

                        • Research & Trends

                          Dive into research and trends in the search industry.

                        • Support

                          Connect on product support and feature requests.

                        • See all categories

                        Related Questions

                        • Alancito

                          Alternate page with proper canonical tag Status: Excluded in Google webmaster tools.

                          canonical google webmaster

                          In Google Webmaster Tools, I have a coverage issue. I am getting this error message: Alternate page with proper canonical tag Status: Excluded. It gives the below blog post page as an example. Any idea how to resolve? At one time, I was using handl utm grabber, but the plugin is deactivated on my website. https://www.savacations.com/turrialba-costa-ricas-garden-city/?utm_source=deleted&utm_medium=deleted&utm_term=deleted&utm_content=deleted&utm_campaign=deleted&gclid=deleted5.

                          Intermediate & Advanced SEO | | Alancito
                          0
                        • Saba.Elahi.M.

                          When creating a sub-domain, does that sub-domain automatically start with the DA of the main domain?

                          We have a website with a high DA and we are considering sub-folder or sub-domain. One of the great benefits of a sub-folder is that we know we get to keep the high DA, is this also the case for sub-domains? Also if you could provide any sources of information that specify this, I can't see to find anything!

                          Intermediate & Advanced SEO | | Saba.Elahi.M.
                          0
                        • CongthanhThe

                          How to find correct schema type

                          Dear Moz members, I m currently working on schema optimizations of my website casinobesty.com which review online casino websites. I have a doubt which schema itemReviewed type I have to use in the review pages. Currently I m using type as "Game" but I m not sure it is correct. "description": "",
                          "itemReviewed": {
                          "@type": "Game",
                          "name": "LeoVegas Casino",
                          "url": "https://casinobesty.com/casino/leovegas-casino/"
                          }, Thank you

                          Intermediate & Advanced SEO | | CongthanhThe
                          1
                        • dailynaukri

                          Duplicate without user-selected canonical excluded

                          We have pdf files uploaded in the media of wordpress and used in our website. As these pdfs are duplicate content of the original publishers, we have marked links to these pdf urls as nofollow. These pages are also disallowed in robots.txt Now, Google Search Console has shown these pages Excluded as "Duplicate without user-selected canonical" As it comes out we cannot use canonical tag with pdf pages so as to point to the original pdf source If we embed a pdf viewer in our website and fetch the pdfs by passing the urls of the original publisher, would the pdfs be still read as text by google and again create duplicate content issue? Another thing, when the pdf expires and is removed, it would lead to 404 error. If we direct our users to the third party website, then it would add up to our bounce rate. What should be the appropriate way to handle duplicate pdfs? Thanks

                          Intermediate & Advanced SEO | | dailynaukri
                          1
                        • vinodh-spintadigital

                          Does having a different sub domain for your Landing Page and Blog affect your overall SEO benefits and Ranking?

                          We have a domain www.spintadigital.com that is hosted with dreamhost and we also have a seperate subdomain blog.spintadigital.com which is hosted in the Ghost platform and we are also using Unbounce landing pages with the sub domain get.spintadigital.com. I wanted to know whether having subdomain like this would affect the traffic metric and ineffect affect the SEO and Rankings of our site.  I think it does not affect the increase in domain authority, but in places like similar web i get different traffic metrics for the different domains.  As far as i can see in many of the metrics these are considered as seperate websites.  We are currently concentrating more on our blogs and wanted to make sure that it does help in the overall domain. We do not have the bandwidth to promote three different websites, and hence need the community's help to understand what is the best option to take this forward.

                          Intermediate & Advanced SEO | | vinodh-spintadigital
                          0
                        • browndoginteractive

                          Avoiding Duplicate Content with Used Car Listings Database: Robots.txt vs Noindex vs Hash URLs (Help!)

                          Hi Guys, We have developed a plugin that allows us to display used vehicle listings from a centralized, third-party database. The functionality works similar to autotrader.com or cargurus.com, and there are two primary components: 1. Vehicle Listings Pages: this is the page where the user can use various filters to narrow the vehicle listings to find the vehicle they want.
                          2. Vehicle Details Pages: this is the page where the user actually views the details about said vehicle. It is served up via Ajax, in a dialog box on the Vehicle Listings Pages. Example functionality:  http://screencast.com/t/kArKm4tBo The Vehicle Listings pages (#1), we do want indexed and to rank. These pages have additional content besides the vehicle listings themselves, and those results are randomized or sliced/diced in different and unique ways. They're also updated twice per day. We do not want to index #2, the Vehicle Details pages, as these pages appear and disappear all of the time, based on dealer inventory, and don't have much value in the SERPs. Additionally, other sites such as autotrader.com, Yahoo Autos, and others draw from this same database, so we're worried about duplicate content. For instance, entering a snippet of dealer-provided content for one specific listing that Google indexed yielded 8,200+ results:  Example Google query. We did not originally think that Google would even be able to index these pages, as they are served up via Ajax. However, it seems we were wrong, as Google has already begun indexing them. Not only is duplicate content an issue, but these pages are not meant for visitors to navigate to directly! If a user were to navigate to the url directly, from the SERPs, they would see a page that isn't styled right. Now we have to determine the right solution to keep these pages out of the index:  robots.txt, noindex meta tags, or hash (#) internal links. Robots.txt Advantages: Super easy to implement Conserves crawl budget for large sites Ensures crawler doesn't get stuck. After all, if our website only has 500 pages that we really want indexed and ranked, and vehicle details pages constitute another 1,000,000,000 pages, it doesn't seem to make sense to make Googlebot crawl all of those pages. Robots.txt Disadvantages: Doesn't prevent pages from being indexed, as we've seen, probably because there are internal links to these pages. We could nofollow these internal links, thereby minimizing indexation, but this would lead to each 10-25 noindex internal links on each Vehicle Listings page (will Google think we're pagerank sculpting?) Noindex Advantages: Does prevent vehicle details pages from being indexed Allows ALL pages to be crawled (advantage?) Noindex Disadvantages: Difficult to implement (vehicle details pages are served using ajax, so they have no tag. Solution would have to involve X-Robots-Tag HTTP header and Apache, sending a noindex tag based on querystring variables, similar to this stackoverflow solution. This means the plugin functionality is no longer self-contained, and some hosts may not allow these types of Apache rewrites (as I understand it) Forces (or rather allows) Googlebot to crawl hundreds of thousands of noindex pages.  I say "force" because of the crawl budget required.  Crawler could get stuck/lost in so many pages, and my not like crawling a site with 1,000,000,000 pages, 99.9% of which are noindexed. Cannot be used in conjunction with robots.txt. After all, crawler never reads noindex meta tag if blocked by robots.txt Hash (#) URL Advantages: By using for links on Vehicle Listing pages to Vehicle Details pages (such as "Contact Seller" buttons), coupled with Javascript, crawler won't be able to follow/crawl these links.  Best of both worlds:  crawl budget isn't overtaxed by thousands of noindex pages, and internal links used to index robots.txt-disallowed pages are gone. Accomplishes same thing as "nofollowing" these links, but without looking like pagerank sculpting (?) Does not require complex Apache stuff Hash (#) URL Disdvantages: Is Google suspicious of sites with (some) internal links structured like this, since they can't crawl/follow them? Initially, we implemented robots.txt--the "sledgehammer solution." We figured that we'd have a happier crawler this way, as it wouldn't have to crawl zillions of partially duplicate vehicle details pages, and we wanted it to be like these pages didn't even exist. However, Google seems to be indexing many of these pages anyway, probably based on internal links pointing to them. We could nofollow the links pointing to these pages, but we don't want it to look like we're pagerank sculpting or something like that. If we implement noindex on these pages (and doing so is a difficult task itself), then we will be certain these pages aren't indexed. However, to do so we will have to remove the robots.txt disallowal, in order to let the crawler read the noindex tag on these pages. Intuitively, it doesn't make sense to me to make googlebot crawl zillions of vehicle details pages, all of which are noindexed, and it could easily get stuck/lost/etc. It seems like a waste of resources, and in some shadowy way bad for SEO. My developers are pushing for the third solution:  using the hash URLs. This works on all hosts and keeps all functionality in the plugin self-contained (unlike noindex), and conserves crawl budget while keeping vehicle details page out of the index (unlike robots.txt). But I don't want Google to slap us 6-12 months from now because it doesn't like links like these (). Any thoughts or advice you guys have would be hugely appreciated, as I've been going in circles, circles, circles on this for a couple of days now. Also, I can provide a test site URL if you'd like to see the functionality in action.

                          Intermediate & Advanced SEO | | browndoginteractive
                          0
                        • Modi

                          Robots Disallow Backslash - Is it right command

                          Bit skeptical, as due to dynamic url and some other linkage issue, google has crawled url with backslash and asterisk character ex - www.xyz.com/\/index.php?option=com_product www.xyz.com/\"/index.php?option=com_product Now %5c is the encoded version of \ - backslash & %22 is encoded version of asterisk Need to know for command :- User-agent: *   Disallow: \As am disallowing all backslash url through this - will it only remove the backslash url which are duplicates or the entire site,

                          Intermediate & Advanced SEO | | Modi
                          0
                        • AndrewY

                          Blocking Dynamic URLs with Robots.txt

                          Background: My e-commerce site uses a lot of layered navigation and sorting links.  While this is great for users, it ends up in a lot of URL variations of the same page being crawled by Google.  For example, a standard category page: www.mysite.com/widgets.html ...which uses a "Price" layered navigation sidebar to filter products based on price also produces the following URLs which link to the same page: http://www.mysite.com/widgets.html?price=1%2C250 http://www.mysite.com/widgets.html?price=2%2C250 http://www.mysite.com/widgets.html?price=3%2C250 As there are literally thousands of these URL variations being indexed, so I'd like to use Robots.txt to disallow these variations. Question: Is this a wise thing to do?  Or does Google take into account layered navigation links by default, and I don't need to worry. To implement, I was going to do the following in Robots.txt: User-agent: * Disallow: /*? Disallow: /*= ....which would prevent any dynamic URL with a '?" or '=' from being indexed.  Is there a better way to do this, or is this a good solution? Thank you!

                          Intermediate & Advanced SEO | | AndrewY
                          1

                        Get started with Moz Pro!

                        Unlock the power of advanced SEO tools and data-driven insights.

                        Start my free trial
                        Products
                        • Moz Pro
                        • Moz Local
                        • Moz API
                        • Moz Data
                        • STAT
                        • Product Updates
                        Moz Solutions
                        • SMB Solutions
                        • Agency Solutions
                        • Enterprise Solutions
                        Free SEO Tools
                        • Domain Authority Checker
                        • Link Explorer
                        • Keyword Explorer
                        • Competitive Research
                        • Brand Authority Checker
                        • MozBar Extension
                        • MozCast
                        Resources
                        • Blog
                        • SEO Learning Center
                        • Help Hub
                        • Beginner's Guide to SEO
                        • How-to Guides
                        • Moz Academy
                        • API Docs
                        About Moz
                        • About
                        • Team
                        • Careers
                        • Contact
                        Why Moz
                        • Case Studies
                        • Testimonials
                        Get Involved
                        • Become an Affiliate
                        • MozCon
                        • Webinars
                        • Practical Marketer Series
                        • MozPod
                        Connect with us

                        Contact the Help team

                        Join our newsletter
                        Moz logo
                        © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                        • Accessibility
                        • Terms of Use
                        • Privacy

                        Looks like your connection to Moz was lost, please wait while we try to reconnect.