Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. On-Page Optimization
    4. How to exclude URL filter searches in robots.txt

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    How to exclude URL filter searches in robots.txt

    On-Page Optimization
    4
    9
    12044
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • neenor
      neenor last edited by

      When I look through my MOZ reports I can see it's included 'pages' which it shouldn't have included i.e. adding filtering rules such as this one http://www.mydomain.com/brands?color=364&manufacturer=505

      How can I exclude all of these filters in the robots.txt? I think it'll be:

      Disallow: /*?color=$

      Is that the correct syntax with the $ sign in it? Thanks!

      1 Reply Last reply Reply Quote 0
      • Everett
        Everett @swissmadesmt last edited by

        Unless you're specifically calling out Bing or Baidu... in your Robots.txt file they should follow the same directives as Google so testing with Google's Robots.txt file tester should suffice for all of them.

        1 Reply Last reply Reply Quote 0
        • swissmadesmt
          swissmadesmt last edited by

          Yes, but what about bing and rest of Search Engine?

          Everett 1 Reply Last reply Reply Quote 0
          • Everett
            Everett @neenor last edited by

            Adrian,

            I agree that there certainly is a right answer to the question posted, as the question asks specifically about one way to manage the issue, being a block of filters in the robots.txt file. What I was getting at is that this may or may not necessarily be the "best" way, and that I'd need to look at your site and your unique situation to figure our which would be the best solution for your needs.

            It is very likely that with these parameters a robots.txt file block is the best approach, assuming the parameters aren't added by default into category page or category pagination page navigational links, as then it would affect the bot's ability to crawl the site. Also, if people are linking to those URLs (highly unlikely though) you may consider a robots meta noindex,follow tag instead so the pagerank could flow to other pages.

            And I'm not entirely sure the code you provided above will work if the blocked parameter is the first one in the string (e.g. domain.com/category/?color=red) as there is the additional wildcard between the ? and the parameter. I would advise testing this in Google Webmaster Tools first.

            1. On the Webmaster Tools Home page, click the site you want.
            2. Under Crawl, click Blocked URLs.
            3. If it's not already selected, click the Test robots.txt tab.
            4. Copy the content of your robots.txt file, and paste it into the first box.
            5. In the URLs box, list the site to test against.
            6. In the User-agents list, select the user-agents you want (e.g. Googlebot)
            1 Reply Last reply Reply Quote 0
            • neenor
              neenor @Everett last edited by

              There certainly is a right answer to my question - I already posted it here earlier today:

              Disallow: /*?color=
               Disallow: /
              ?*manufacturer=

              Without the $ at the end which would otherwise denote the end of the URL.

              Everett 1 Reply Last reply Reply Quote -1
              • Everett
                Everett last edited by

                Hello Adrian,

                The Moz reports are meant to help you uncover issues like this. If you're seeing non-canonical URLs in the Moz report then there is a potential issue for Google, Bing and other search engines as well.

                Google does respect wildcards (*) in the robots.txt file, though it can easily be done wrong. There is not right or wrong answer to the issue of using filters or faceted navigation, as each circumstance is going to be different. However, I hope some of these articles will help you identify the best approach for your needs:

                (Note: Faceted Navigation is not exactly the same as category filters, but the issues and possible solutions are very similar

                )Building Faceted Navigation That Doesn't Suck Faceted Navigation Whiteboard Friday
                Duplicate Content: Block, Redirect or Canonical
                Guide to eCommerce Facets, Filters and Categories
                Rel Canonical How To and Why Not
                Moz.com Guide to Duplicate Content

                I don't know how your store handles these (e.g. does it add the filter automatically, or only when a user selects a filter?) so I can't give you the answer, but I promise if you read those articles above you will have a very good understanding of all of the options so you can choose which is best for you. That might end up being as simple as blocking the filters in your robots.txt file, or you may opt for rel canonical, noindex meta tag, ajax, Google parameter handling, etc...

                Good luck!

                neenor 1 Reply Last reply Reply Quote 0
                • neenor
                  neenor @stever999 last edited by

                  It's not Google's index that I'm interested in in this case, it's for the MOZ reports. Moz was including over 10,000 'pages' because it was indexing these URLs. Now I know how to edit the robots.txt Moz will be prevented from indexing them again (we only have around 2,000 real pages, not 10,000)

                  1 Reply Last reply Reply Quote 0
                  • neenor
                    neenor last edited by

                    I sought out the answer from a developer and got the following reply, so posting here in case it helps someone else:

                    To exclude pages with color or manufacture in them you can use

                    Disallow: /*?color=
                     Disallow: /
                    ?*manufacturer=

                    A  question mark in your try should be omitted as it denotes the end of the url

                    1 Reply Last reply Reply Quote 0
                    • stever999
                      stever999 last edited by

                      Hi

                      I would recommend excluding these in Google Webmaster Tools. Once logged in to your account under the "Crawl" menu you will find "URL Parameters". Find the relevant parameter in the list on this page and you can tell Google not to index these pages.

                      Hope this helps.

                      Steve

                      neenor 1 Reply Last reply Reply Quote 0
                      • 1 / 1
                      • First post
                        Last post

                      Got a burning SEO question?

                      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                      Start my free trial


                      Browse Questions

                      Explore more categories

                      • Moz Tools

                        Chat with the community about the Moz tools.

                      • SEO Tactics

                        Discuss the SEO process with fellow marketers

                      • Community

                        Discuss industry events, jobs, and news!

                      • Digital Marketing

                        Chat about tactics outside of SEO

                      • Research & Trends

                        Dive into research and trends in the search industry.

                      • Support

                        Connect on product support and feature requests.

                      • See all categories

                      Related Questions

                      • Cricket93

                        Ranking Expectations and Search Intent Alignment

                        keyword ranking search intent

                        I've recently implemented the page optimization recommendations provided by Moz Pro to help our site rank for specific keywords on certain pages. It’s been about two weeks since we've added these URLs/keyword parings. The optimization scores for the targeted keywords and URLs are looking strong. Also, we've crafted these pages with keyword optimisation in mind. However, we have a couple of questions as we move forward: Ranking Expectations: Since our optimization scores are good, when might we start seeing improvements in our keyword rankings? We know that SEO results can take time, but we would appreciate any insights on a typical timeline based on your experiences. Optimization and Search Intent: While the tool’s optimization suggestions have been helpful in regards to giving us a score for a specific keyword, we’re curious about how this factors into search intent. How does this tool take into account variations in search intent, especially if users search using slight variations of the keywords we’re targeting? Thank you so much for your insight!

                        On-Page Optimization | | Cricket93
                        0
                      • Mike555

                        Solved Recommended title length for Google search results

                        I read the recommended title length is 50-60 characters depending on alphabets, etc,.
                        Anyways, my question is, is there any harm of having longer title?
                        If all my important keywords are within the 50-60 characters that will show up on search results, I can still make the title longer, it's just that those keywords outside won't have any effect on search results?

                        On-Page Optimization | | Mike555
                        0
                      • maghanlinchpinsales

                        City Name in URL structure

                        I have a client whose site was built when they only served one market, and they now have that city in the majority of their URLs. I'm suggesting we redo the URL structure to remove this location from the main URLs (think homepage, about, etc.) since they have now expanded to three markets. They are seeing a lot of great organic traffic in that original market but are struggling in the new ones they've added so I'm helping to optimize their site. How critical do you think that removing that location from the URL is? I know we would need to implement 301 redirects, but wanted to get thoughts on this.

                        On-Page Optimization | | maghanlinchpinsales
                        0
                      • Faye234

                        Why do I have 2 different URL's for the same page - is this good practice?

                        Hi GuysMy father is currently using a programmer to build his new site. Knowing a little about SEO etc, I was a little suspicious of the work carried out. **Anyone with good programming and SEO knowledge, please offer your advice!**This page http://www.thewoodgalleries.co.uk/gallery-range-wood-flooring/ which is soon to be http://www.thewoodgalleries.co.uk/engineered-wood/ you'll see has a number of different products. The products on this particular page have been built into colour categories like thishttp://www.thewoodgalleries.co.uk/engineered-wood/lights-greys http://www.thewoodgalleries.co.uk/engineered-wood/beiges http://www.thewoodgalleries.co.uk/engineered-wood/browns http://www.thewoodgalleries.co.uk/engineered-wood/darks-blacks This is fine. Eventually when we add to our selection of woods, we'll easily segment each product into "colour categories" for users to easily navigate to. My question is  - Why do I have 2 different URL's for the same page - is this good practice? Please see below... Visible URL - http://www.thewoodgalleries.co.uk/engineered-wood/browns/cipressa/Below is the permalink seen in Word Press for this page also.Permalink: http://www.thewoodgalleries.co.uk/engineered-wood/browns-engineered-wood/cipressa/and in the Word Press snippet shows the same permalink urlCipressa | Engineered Brown Wood | The Wood Gallerieswww.thewoodgalleries.co.uk/engineered-wood/browns-engineered-wood/cipressa/ Buy Cipressa Engineered Brown Wood, available at The Wood Galleries, London. Provides an Exceptional Foundation for Elegant Décor, Extravagant .. If this is completely ok and has no negative search impact - then I'm happy. If not what should I advise to my programmer to do? Your help would be very much appreciated. Regards Faye

                        On-Page Optimization | | Faye234
                        0
                      • SeoMartin1

                        URLs and folder structure for an E-commerce

                        Hi there !-) I´m helping a friend who has a e-commerce about nail polish in Brazil. I´m a little in doubt about the urls and folder structure. Two questions: 1. There are 10 products per category and 50 categories. Should I put them all in the root folder or creat 2 major categories ( 25 sub-categories each one)? 2. Whats the better product page url ( the store has around 500) nailpolish.com/IMPORT/BRAND/NAME-OF-THE-PRODUCT OR nailpolish.com/COMPLETE-NAME-OF-THE-PRODUCT Whats the best recomandation?

                        On-Page Optimization | | SeoMartin1
                        0
                      • Switch_Digital

                        URL length... is >115 now >255?

                        I've been having detailed discussions with a CMS provider on behalf of a client. Long URLs are the least of their problems however, the developer is arguing that Google has amended their algorithm and will now read URLs that are up to 255 characters long. I have stated that as far as I am aware, Google will still not read URLs over 115 characters... Before I stamp my feet, can someone confirm what is actually happening? SEOmoz still classes URLs >115 characters long as an amber issue. Thanks

                        On-Page Optimization | | Switch_Digital
                        0
                      • CommercePundit

                        How to Define Best URL Structure for Product Pages?

                        I am working on my website to edit structure with help of Google's search engine optimization starter guide. There is really good instruction to define URL structure which help us to perform well over Google's organic search. I have resolved issues regarding category pages but, I have confusion to define best URL structure for product pages. My website's product page URL structure is as follow. http://www.vistastores.com/marketumbrellas-californiaumbrella-slpt758-f13-red.html http://www.vistastores.com/homefurniture-winsomewood-93630.html URL structure is constructed with following terms. 1. Root Category Name (Market Umbrellas or Home Furniture or ....) 2. Brand Name 3. Manufacturer Part Number I am not happy with this structure and also not performing well over Google's organic search. I am thinking to include product name or title tag in URL after root domain. But, it may create very long URL and create issues in organic search display. Does it really matter to perform well over Google's organic search? How can I define best URL structure for product pages?

                        On-Page Optimization | | CommercePundit
                        0
                      • matt-14567

                        Does it matter if your URL ends in .net or .com?

                        Someone told me that having a URL that ends in .net (instead of .com) will hurt my site's SEO. Is that true?

                        On-Page Optimization | | matt-14567
                        0

                      Get started with Moz Pro!

                      Unlock the power of advanced SEO tools and data-driven insights.

                      Start my free trial
                      Products
                      • Moz Pro
                      • Moz Local
                      • Moz API
                      • Moz Data
                      • STAT
                      • Product Updates
                      Moz Solutions
                      • SMB Solutions
                      • Agency Solutions
                      • Enterprise Solutions
                      Free SEO Tools
                      • Domain Authority Checker
                      • Link Explorer
                      • Keyword Explorer
                      • Competitive Research
                      • Brand Authority Checker
                      • Local Citation Checker
                      • MozBar Extension
                      • MozCast
                      Resources
                      • Blog
                      • SEO Learning Center
                      • Help Hub
                      • Beginner's Guide to SEO
                      • How-to Guides
                      • Moz Academy
                      • API Docs
                      About Moz
                      • About
                      • Team
                      • Careers
                      • Contact
                      Why Moz
                      • Case Studies
                      • Testimonials
                      Get Involved
                      • Become an Affiliate
                      • MozCon
                      • Webinars
                      • Practical Marketer Series
                      • MozPod
                      Connect with us

                      Contact the Help team

                      Join our newsletter
                      Moz logo
                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                      • Accessibility
                      • Terms of Use
                      • Privacy

                      Looks like your connection to Moz was lost, please wait while we try to reconnect.