The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Forum
    2. Categories
    3. SEO Tactics
    4. Intermediate & Advanced SEO
    5. Robots.txt: Can you put a /* wildcard in the middle of a URL?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Robots.txt: Can you put a /* wildcard in the middle of a URL?

    Intermediate & Advanced SEO
    2 2 1.1k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • IHSwebsite
      IHSwebsite last edited by

      We have noticed that Google is indexing the language/country directory versions of directories we have disallowed in our robots.txt.

      For example:

      Disallow: /images/ is blocked just fine

      However, once you add our /en/uk/ directory in front of it, there are dozens of pages indexed.

      The question is: Can I put a wildcard in the middle of the string, ex. /en/*/images/, or do I need to list out every single country for every language in the robots file. Anyone know of any workarounds?

      1 Reply Last reply Reply Quote 0
      • irvingw
        irvingw last edited by

        Yes, wildcards work, thank god.

        1 Reply Last reply Reply Quote 1
        • 1 / 1
        • First post
          Last post

        Got a burning SEO question?

        Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


        Start my free trial


        Explore more categories

        • Moz Tools

          Chat with the community about the Moz tools.

          Getting Started
          Moz Pro
          Moz Local
          Moz Bar
          API
          What's New

        • SEO Tactics

          Discuss the SEO process with fellow marketers

          Content Development
          Competitive Research
          Keyword Research
          Link Building
          On-Page Optimization
          Technical SEO
          Reporting & Analytics
          Intermediate & Advanced SEO
          Image & Video Optimization
          International SEO
          Local SEO

        • Community

          Discuss industry events, jobs, and news!

          Moz Blog
          Moz News
          Industry News
          Jobs and Opportunities
          SEO Learn Center
          Whiteboard Friday

        • Digital Marketing

          Chat about tactics outside of SEO

          Affiliate Marketing
          Branding
          Conversion Rate Optimization
          Web Design
          Paid Search Marketing
          Social Media

        • Research & Trends

          Dive into research and trends in the search industry.

          SERP Trends
          Search Behavior
          Algorithm Updates
          White Hat / Black Hat SEO
          Other SEO Tools

        • Support

          Connect on product support and feature requests.

          Product Support
          Feature Requests
          Participate in User Research

        • See all categories

        • What does Disallow: /french-wines/?* actually do - robots.txt
          McTaggart
          McTaggart
          0
          8
          1.8k

        • If I block a URL via the robots.txt - how long will it take for Google to stop indexing that URL?
          Gabriele_Layoutweb
          Gabriele_Layoutweb
          0
          3
          821

        • Will disallowing URL's in the robots.txt file stop those URL's being indexed by Google
          andyheath
          andyheath
          0
          10
          3.2k

        • Wildcarding Robots.txt for Particular Word in URL
          EvansHunt
          EvansHunt
          0
          6
          2.1k

        • Blocking Dynamic URLs with Robots.txt
          AndrewY
          AndrewY
          1
          4
          6.9k

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy

        Looks like your connection to Moz was lost, please wait while we try to reconnect.