Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      Track AI Overviews in Keyword Research
      Moz Pro

      Track AI Overviews in Keyword Research

      Try it free!
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. On-Page Optimization
    4. How do i block an entire category/directory with robots.txt?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    How do i block an entire category/directory with robots.txt?

    On-Page Optimization
    2
    3
    4237
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • bricerhodes
      bricerhodes last edited by

      Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.

      The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.

      I'm not really sure how i'd type it into the robots.txt file, or where to place the file.

      any help would be appreciated thanks

      1 Reply Last reply Reply Quote 0
      • bricerhodes
        bricerhodes last edited by

        Thanks for the detailed answer, i will give it a try!

        1 Reply Last reply Reply Quote 0
        • LynnPatchett
          LynnPatchett last edited by

          Hi

          This should do it, you place the robots.txt in the root directory of your site.

          User-agent: * Disallow: /product-category/

          You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt

          As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.

          Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).

          It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.

          Hope that helps!

          1 Reply Last reply Reply Quote 3
          • 1 / 1
          • First post
            Last post

          Got a burning SEO question?

          Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


          Start my free trial


          Browse Questions

          Explore more categories

          • Moz Tools

            Chat with the community about the Moz tools.

          • SEO Tactics

            Discuss the SEO process with fellow marketers

          • Community

            Discuss industry events, jobs, and news!

          • Digital Marketing

            Chat about tactics outside of SEO

          • Research & Trends

            Dive into research and trends in the search industry.

          • Support

            Connect on product support and feature requests.

          • See all categories

          Related Questions

          • RosemaryB

            Can we change Title and/or Descriptions Dynamically Based on Search Query

            If I recall we used to be able to change our title attributes tag dynamically based on the search query but not sure if it's possible now or if it makes sense to do so. Thoughts? Rosemary

            On-Page Optimization | | RosemaryB
            1
          • Lorne_Marr

            Removing navigation menu items/links on homepage

            We are redesigning our website after a long stint with an SEO firm who also handled our design/dev. We want to clean up the links on our homepage but don't want to screw up our IA or SEO. We want to delete some navbar menu items and a whole bunch on random links to our evergreen content below the fold. Would we need to reposition those navbar items/content links to our footer or somewhere else on the homepage to maintain our internal linking structure? It would be great if you could take a look at our site and give us any suggestions or advice on the best way to go about this. Thanks!

            On-Page Optimization | | Lorne_Marr
            1
          • Shalin.TJ

            Tags vs. Categories? What should I use?

            I'm starting with a blog (self-hosted wordpress) and I'm thinking of the following content structure so that the readers are easily able to locate relevant content: Background: It's a blog which gives people relevant info about government jobs. To start with we will just be publishing information about these jobs but over a period of time also intend to post content that helps readers prepare for these jobs. In other words, right now it's just about detailed job notifications but in the coming months, we shall also post about preparation-related information. Typically, each of the job notifications can be bifurcated like: Jobs basis industry Banking Railways Clinical, etc. Jobs basis company ABC co. DEF co. XYZ co. etc. Jobs basis State / City City 1 City 2, etc. Jobs basis educational qualification Graduation Post-Graduation, etc. Now, I'm seriously confused how should I structure this data from the perspective of Categories & Tags such that it's reader as well as SEO-friendly. Do note that each of the government jobs post ideally falls in a couple of above mentioned categories. Thanks..

            On-Page Optimization | | Shalin.TJ
            0
          • lbohen

            State Abbreviations and/or Fully Spelled Out?

            Should State names be abbreviated and/or fully spelled out in title tags, meta descriptions and body content? Does Google know that VT = Vermont?

            On-Page Optimization | | lbohen
            0
          • mihaiaperghis

            WordPress and category/subcategory landing pages

            Hey, Here's my situation. I'm building a WordPress blog for product reviews of a certain niche. Current category setup is 4 main categories with 4-8 subcategories each. Each subcategory has a unique description that will help it become a landing page for certain keywords, after which it lists the posts from that subcategory. The posts will always be assigned to a sub-category, never to a main category. My issue is what to do with the main categories. They're fairly general so they're not really targeting any keywords, and don't have any unique descriptions attached to them. I was thinking of choosing between three options on designing the main category pages: List the subcategories + normal posts loop that bring the latest posts from the subcategories (may create a lot of duplicate content since the subcategory pages are also listing their posts) List only the subcategories (+ maybe just the latest post from each subcategory) Don't link the main categories at all, instead only use them to create dropdowns for the subcategories So, what would you choose, and why?

            On-Page Optimization | | mihaiaperghis
            0
          • editabletext

            Does Google index dynamically generated content/headers, etc.?

            To avoid dupe content, we are moving away from a model where we have 30,000 pages, each with a separate URL that looks like /prices/<product-name>/<city><state>,  often with dupe content because the product overlaps from city to city, and it's hard to keep 30,000 pages unique, where sometimes the only distinction is the price & the city/state.</state></city></product-name> We are moving to a model with around 300 unique pages, where some of the info that used to be in the url will move to the page itself (headers, etc.) to cut down on  dupe content on those unique 300 pages. My question is this. If we have 300 unique-content pages with unique URL's, and we then put some dynamic info (year, city, state) into the page itself, will Google index this dynamic content? The question behind this one is, how do we continue to rank for searches for that product in the city-state being searched without having that info in the URL? Any best practices we should know about?

            On-Page Optimization | | editabletext
            0
          • KMack

            Wordpress: Should I NO INDEX Categories & Archives Pages?

            I am new to SEOmoz & trying to work my way through the ca-trillion errors that have been found on my site, but for each one I want to ensure that I am helping rather than harming my site. The tool has (as a "notice") said that my category pages & Archives are NO-INDEX, is this how these pages should be dealt with? In addition, the crawler has also (as a "warning error) discovered that my categories, and Archives do not have a meta description..is this of great importance for non indexed pages of this type? Thanks so much to the SEOmoz forum members, you have so far been of invaluable help to me.

            On-Page Optimization | | KMack
            2
          • eyepaq

            301 redirect (www.domain.com/index to www.domain.com)

            Hello, Please let me know what are the exact right steps in order to get rid of the duplicate content issues related with: www.domain.com/index.html same as www.domain.com  without creating an infinite loop. Do you have a step by step guide posted within seomoz including 301 redirect for non www to www for all urls and index.whatever to main domain name without going into a infinite loop ? btw how to you spot the loop ? is it obvious like never ending refresh of the home page ? thanks a lot !

            On-Page Optimization | | eyepaq
            2

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          • Digital Marketers
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy

          Looks like your connection to Moz was lost, please wait while we try to reconnect.