Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Intermediate & Advanced SEO
    4. Robots.txt: how to exclude sub-directories correctly?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Robots.txt: how to exclude sub-directories correctly?

    Intermediate & Advanced SEO
    3
    10
    53113
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • fablau
      fablau last edited by

      Hello here,

      I am trying to figure out the correct way to tell SEs to crawls this:

      http://www.mysite.com/directory/

      But not this:

      http://www.mysite.com/directory/sub-directory/

      or this:

      http://www.mysite.com/directory/sub-directory2/sub-directory/...

      But with the fact I have thousands of sub-directories with almost infinite combinations, I can't put the following definitions in a manageable way:

      disallow: /directory/sub-directory/

      disallow: /directory/sub-directory2/

      disallow: /directory/sub-directory/sub-directory/

      disallow: /directory/sub-directory2/subdirectory/

      etc...

      I would end up having thousands of definitions to disallow all the possible sub-directory combinations.

      So, is the following way a correct, better and shorter way to define what I want above:

      allow: /directory/$

      disallow: /directory/*

      Would the above work?

      Any thoughts are very welcome! Thank you in advance.

      Best,

      Fab.

      1 Reply Last reply Reply Quote 1
      • MickEdwards
        MickEdwards @sjunaidali last edited by

        I mentioned both.  You add a meta robots to noindex and remove from the sitemap.

        1 Reply Last reply Reply Quote 0
        • sjunaidali
          sjunaidali @MickEdwards last edited by

          But google is still free to index a link/page even if it is not included in xml sitemap.

          MickEdwards 1 Reply Last reply Reply Quote 0
          • MickEdwards
            MickEdwards @sjunaidali last edited by

            Install Yoast Wordpress SEO plugin and use that to restrict what is indexed and what is allowed in a sitemap.

            sjunaidali 1 Reply Last reply Reply Quote 1
            • sjunaidali
              sjunaidali @MickEdwards last edited by

              I am using wordpress, Enfold theme (themeforest).

              I want some files to be accessed by google, but those should not be indexed.

              Here is an example: http://prntscr.com/h8918o

              I have currently blocked some JS directories/files using robots.txt (check screenshot)

              But due to this I am not able to pass Mobile Friendly Test on Google: http://prntscr.com/h8925z (check screenshot)

              Is its possible to allow access, but use a tag like noindex in the robots.txt file. Or is there any other way out.

              MickEdwards 1 Reply Last reply Reply Quote 0
              • fablau
                fablau last edited by

                Yes, everything looks good, Webmaster Tools gave me the expected results with the following directives:

                allow: /directory/$

                disallow: /directory/*

                Which allows this URL:

                http://www.mysite.com/directory/

                But doesn't allow the following one:

                http://www.mysite.com/directory/sub-directory2/...

                This page also gives an update similar to mine:

                https://support.google.com/webmasters/answer/156449?hl=en

                I think I am good! Thanks 🙂

                1 Reply Last reply Reply Quote 2
                • fablau
                  fablau last edited by

                  Thank you Michael, it is my understanding then that my idea of doing this:

                  allow: /directory/$

                  disallow: /directory/*

                  Should work just fine. I will test it within Google Webmaster Tools, and let you know if any problems arise.

                  In the meantime if anyone else has more ideas about all this and can confirm me that would be great!

                  Thank you again.

                  1 Reply Last reply Reply Quote 1
                  • MickEdwards
                    MickEdwards @fablau last edited by

                    I've always stuck to Disallow and followed -

                    "This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory:"

                    http://www.robotstxt.org/robotstxt.html

                    From https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt this seems contradictory

                    | /* | equivalent to / | equivalent to / | Equivalent to "/" -- the trailing wildcard is ignored. |

                    I think this post will be very useful  for you - http://moz.com/community/q/allow-or-disallow-first-in-robots-txt

                    1 Reply Last reply Reply Quote 1
                    • fablau
                      fablau @MickEdwards last edited by

                      Thank you Michael,

                      Google and other SEs actually recognize the "allow:" command:

                      https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

                      The fact is: if I don't specify that, how can I be sure that the following single command:

                      disallow: /directory/*

                      Doesn't prevent SEs to spider the /directory/ index as I'd like to?

                      MickEdwards 1 Reply Last reply Reply Quote 0
                      • MickEdwards
                        MickEdwards last edited by

                        As long as you dont have directories somewhere in /* that you want indexed then I think that will work.  There is no allow so you don't need the first line just

                        disallow: /directory/*

                        You can test out here- https://support.google.com/webmasters/answer/156449?rd=1

                        fablau sjunaidali 2 Replies Last reply Reply Quote 0
                        • 1 / 1
                        • First post
                          Last post

                        Got a burning SEO question?

                        Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                        Start my free trial


                        Browse Questions

                        Explore more categories

                        • Moz Tools

                          Chat with the community about the Moz tools.

                        • SEO Tactics

                          Discuss the SEO process with fellow marketers

                        • Community

                          Discuss industry events, jobs, and news!

                        • Digital Marketing

                          Chat about tactics outside of SEO

                        • Research & Trends

                          Dive into research and trends in the search industry.

                        • Support

                          Connect on product support and feature requests.

                        • See all categories

                        Related Questions

                        • joshibhargav_20

                          Using a Reverse Proxy and 301 redirect to appear Sub Domain as Sub Directory - what are the SEO Risks?

                          We’re in process to move WordPress blog URLs from subdomains to sub-directory. We aren’t moving blog physically, but using reverse proxy and 301 redirection to do this. Blog subdomain URL is https://blog.example.com/ and destination sub-directory URL is https://www.example.com/blog/ Our main website is e-commerce marketplace which is YMYL site. This is on Windows server. Due to technical reasons, we can’t physically move our WordPress blog to the main website. Following is our Technical Setup Setup a reverse proxy at https://www.example.com/blog/ pointing to https://blog.example.com/ Use a 301 redirection from https://blog.example.com/ to https://www.example.com/blog/ with an exception if a traffic is coming from main WWW domain then it won’t redirect. Thus, we can eliminate infinite loop. Change all absolute URLs to relative URLs on blog Change the sitemap URL from https://blog.example.com/sitemap.xml to https://www.example.com/blog/sitemap.xml and update all URLs mentioned within the sitemap. SEO Risk Evaluation We have individual GA Tracking ID and individual Google Search Console Properties for main website and blog. We will not merge them. Keep them separate as they are. Keeping this in mind, I am evaluating SEO Risks factors Right now when we receive traffic from main website to blog (or vice versa) then it is considered as referral traffic and new cookies are set for Google Analytics. What’s going to happen when its on the same domain? Which type of settings change should I do in Blog’s Google Search Console? (A). Do I need to request “Change of Address” in the Blog’s search console property? (B). Should I re-submit the sitemap? Do I need to re-submit the blog sitemap from the https://www.example.com/ Google Search Console Property? Main website is e-commerce marketplace which is YMYL website, and blog is all about content. So does that impact SEO? Will this dilute SEO link juice or impact on the main website ranking because following are the key SEO Metrices. (A). Main website’s Avg Session Duration is about 10 minutes and bounce rate is around 30% (B). Blog’s Avg Session Duration is 33 seconds and bounce rate is over 92%

                          Intermediate & Advanced SEO | | joshibhargav_20
                          0
                        • Dan-Louis

                          URL Structure & Best Practice when Facing 4+ Sub-levels

                          Hi. I've spent the last day fiddling with the setup of a new URL structure for a site, and I can't "pull the trigger" on it. Example: - domain.com/games/type-of-game/provider-name/name-of-game/ Specific example: - arcade.com/games/pinball/deckerballs/starshooter2k/ The example is a good description of the content that I have to organize. The aim is to a) define url structure, b) facilitate good ux, **c) **create a good starting point for content marketing and SEO, avoiding multiple / stuffing keywords in urls'. The problem? Not all providers have the same type of game. Meaning, that once I get past the /type-of-game/, I must write a new category / page / content for /provider-name/. No matter how I switch the different "sub-levels" around in the url, at one point, the provider-name doesn't fit as its in need of new content, multiple times. The solution? I can skip "provider-name". The caveat though is that I lose out on ranking for provider keywords as I don't have a cornerstone content page for them. Question: Using the URL structure as outlined above in WordPress, would you A) go with "Pages", or B) use "Posts"

                          Intermediate & Advanced SEO | | Dan-Louis
                          0
                        • Tylerj

                          Should I use noindex or robots to remove pages from the Google index?

                          I have a Magento site and just realized we have about 800 review pages indexed. The /review directory is disallowed in robots.txt but the pages are still indexed. From my understanding robots means it will not crawl the pages BUT if the pages are still indexed if they are linked from somewhere else. I can add the noindex tag to the review pages but they wont be crawled. https://www.seroundtable.com/google-do-not-use-noindex-in-robots-txt-20873.html Should I remove the robots.txt and add the noindex? Or just add the noindex to what I already have?

                          Intermediate & Advanced SEO | | Tylerj
                          0
                        • khi5

                          "noindex, follow" or "robots.txt" for thin content pages

                          Does anyone have any testing evidence what is better to use for pages with thin content, yet important pages to keep on a website? I am referring to content shared across multiple websites (such as e-commerce, real estate etc). Imagine a website with 300 high quality pages indexed and 5,000 thin product type pages, which are pages that would not generate relevant search traffic. Question goes: Does the interlinking value achieved by "noindex, follow" outweigh the negative of Google having to crawl all those "noindex" pages? With robots.txt one has Google's crawling focus on just the important pages that are indexed and that may give ranking a boost. Any experiments with insight to this would be great. I do get the story about "make the pages unique", "get customer reviews and comments" etc....but the above question is the important question here.

                          Intermediate & Advanced SEO | | khi5
                          0
                        • Bio-RadAbs

                          Can you redirect specific sub domain URLs?

                          ello! We host our PDFs, Images, CSS all in a sub domain. For the question, let's call this sub.cyto.com. I've noticed a particular PDF doing really well, infact it has gathered valuable external links from high authoritative sites. To top it off, it gets good visits. I've been going back and forth with our developers to move this PDF to a subfolder structure.
                          For example: www.cyto.com/document/xxxx.pdf In my perspective, if I move this and set up a permanent redirect, then all the external links the PDF gathered, link juice and future visits will be attributed to the main website. Since the PDF is existing in the subdomain, I can't even track direct visits nor get the link juice. It appears in top position of Google as well. My developer says it is better to keep images, pdf, css in the subdomain. I see his point and an idea I have is to: convert the pdf to a webpage. Set up a 301 redirect from the existing subdomain to this webpage Upload the pdf with a new name and link to it from the webpage, so users can download if they choose to. This should give me the existing rank juice. However, my question is whether you can set up a 301 redirect for just a single subdomain URL to a folder structure URL? sub.cyto.com/xxx.pdf to www.cyto.com/document/xxxx.pdf?

                          Intermediate & Advanced SEO | | Bio-RadAbs
                          0
                        • gregelwell

                          Could you use a robots.txt file to disalow a duplicate content page from being crawled?

                          A website has duplicate content pages to make it easier for users to find the information from a couple spots in the site navigation. Site owner would like to keep it this way without hurting SEO. I've thought of using the robots.txt file to disallow search engines from crawling one of the pages. Would you think this is a workable/acceptable solution?

                          Intermediate & Advanced SEO | | gregelwell
                          0
                        • HD_Leona

                          Blocking Pages Via Robots, Can Images On Those Pages Be Included In Image Search

                          Hi! I have pages within my forum where visitors can upload photos.  When they upload photos they provide a simple statement about the photo but no real information about the image,definitely not enough for the page to be deemed worthy of being indexed.  The industry however is one that really leans on images and having the images in Google Image search is important to us. The url structure is like such:  domain.com/community/photos/~username~/picture111111.aspx I wish to block the whole folder from Googlebot to prevent these low quality pages from being added to Google's main SERP results.  This would be something like this: User-agent: googlebot Disallow: /community/photos/ Can  I disallow Googlebot specifically rather than just using User-agent:  * which would then allow googlebot-image to pick up the photos?  I plan on configuring a way to add meaningful alt attributes and image names to assist in visibility, but the actual act of blocking the pages and getting the images picked up... Is this possible? Thanks! Leona

                          Intermediate & Advanced SEO | | HD_Leona
                          0
                        • seo123456

                          Using 2 wildcards in the robots.txt file

                          I have a URL string which I don't want to be indexed. it includes the characters _Q1 ni the middle of the string. So in the robots.txt can I use 2 wildcards in the string to take out all of the URLs with that in it?  So something like /_Q1.  Will that pickup and block every  URL with those characters in the string? Also, this is not directly of the root, but in a secondary directory, so .com/.../_Q1.  So do I have to format the robots.txt as //_Q1* as it will be in the second folder or just using /_Q1 will pickup everything no matter what folder it is on? Thanks.

                          Intermediate & Advanced SEO | | seo123456
                          0

                        Get started with Moz Pro!

                        Unlock the power of advanced SEO tools and data-driven insights.

                        Start my free trial
                        Products
                        • Moz Pro
                        • Moz Local
                        • Moz API
                        • Moz Data
                        • STAT
                        • Product Updates
                        Moz Solutions
                        • SMB Solutions
                        • Agency Solutions
                        • Enterprise Solutions
                        Free SEO Tools
                        • Domain Authority Checker
                        • Link Explorer
                        • Keyword Explorer
                        • Competitive Research
                        • Brand Authority Checker
                        • Local Citation Checker
                        • MozBar Extension
                        • MozCast
                        Resources
                        • Blog
                        • SEO Learning Center
                        • Help Hub
                        • Beginner's Guide to SEO
                        • How-to Guides
                        • Moz Academy
                        • API Docs
                        About Moz
                        • About
                        • Team
                        • Careers
                        • Contact
                        Why Moz
                        • Case Studies
                        • Testimonials
                        Get Involved
                        • Become an Affiliate
                        • MozCon
                        • Webinars
                        • Practical Marketer Series
                        • MozPod
                        Connect with us

                        Contact the Help team

                        Join our newsletter
                        Moz logo
                        © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                        • Accessibility
                        • Terms of Use
                        • Privacy

                        Looks like your connection to Moz was lost, please wait while we try to reconnect.