Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      What is your Brand Authority?
      Moz

      What is your Brand Authority?

      Check yours now
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. Product Support
    3. How to block Rogerbot From Crawling UTM URLs

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    How to block Rogerbot From Crawling UTM URLs

    Product Support
    4
    7
    2951
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • Firestarter-SEO
      Firestarter-SEO Subscriber last edited by

      I am trying to block roger from crawling some UTM urls we have created, but having no luck. My robots.txt file looks like:

      User-agent: rogerbot
      Disallow: /?utm_source*
      
      This does not seem to be working. Any ideas? 
      
      1 Reply Last reply Reply Quote 0
      • tawnycase
        tawnycase @Jenny1 last edited by

        Shoot! There may be something else going on. Give us a shout at help@moz.com and we'll see if we can figure it out!

        1 Reply Last reply Reply Quote 0
        • Jenny1
          Jenny1 Subscriber @tawnycase last edited by

          FYI - I tried this and it did not work. Rogerbot is still picking up URL's we don't need. It's making my crawl report a mess!

          tawnycase 1 Reply Last reply Reply Quote 0
          • tawnycase
            tawnycase @Firestarter-SEO last edited by

            The only difference there is the * wildchar. The string with that character will limit the crawler from accessing any URL with that string of characters in it. 🙂

            1 Reply Last reply Reply Quote 0
            • Firestarter-SEO
              Firestarter-SEO Subscriber @LoganRay last edited by

              What is the difference between Disallow: /*?utm_ and Disallow: /?utm_  ?

              tawnycase 1 Reply Last reply Reply Quote 0
              • tawnycase
                tawnycase last edited by

                Hi there! Tawny from the Customer Support team here!

                You should be able to add a disallow directive for that parameter and any others to block our crawler from accessing them. It would look something like this:

                User-agent: Rogerbot
                Disallow: ?utm

                etc., until you have blocked all of the parameters that may be causing these duplicate content errors. It looks like the _source* might be what's giving our tools some trouble. It looks like Logan Ray has made an excellent suggestion - give that formatting a try and see if it helps! 🙂

                You can also use the wild card user-agent * in order to block all crawlers from those pages, if you prefer. Here is a great resource about the robots.txt file that might be helpful: https://moz.com/learn/seo/robotstxt We always recommend checking your robots.txt file with a handy Robots Checker Tool once you make changes to avoid any nasty surprises. 🙂

                Jenny1 1 Reply Last reply Reply Quote 0
                • LoganRay
                  LoganRay last edited by

                  Skyler,

                  You're close, give this a shot:

                  Disallow: /*?utm_

                  This will be inclusive of all UTM tags regardless of what comes before the tag or what element you have first.

                  Firestarter-SEO 1 Reply Last reply Reply Quote 3
                  • 1 / 1
                  • First post
                    Last post

                  Got a burning SEO question?

                  Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                  Start my free trial


                  Browse Questions

                  Explore more categories

                  • Moz Tools

                    Chat with the community about the Moz tools.

                  • SEO Tactics

                    Discuss the SEO process with fellow marketers

                  • Community

                    Discuss industry events, jobs, and news!

                  • Digital Marketing

                    Chat about tactics outside of SEO

                  • Research & Trends

                    Dive into research and trends in the search industry.

                  • Support

                    Connect on product support and feature requests.

                  • See all categories

                  Related Questions

                  • Azurius

                    Unsolved Crawling only the Home of my website

                    crawl error crawling crawl

                    Hello,
                    I don't understand why MOZ crawl only the homepage of our webiste https://www.modelos-de-curriculum.com We add the website correctly, and we asked for crawling all the pages. But the tool find only the homepage. Why? We are testing the tool before to suscribe. But we need to be sure that the tool is working for our website. If you can please help us.

                    Product Support | | Azurius
                    0
                  • ghrisa65

                    Unsolved 403 crawl error

                    403 errors

                    Hi, Moz( Also reported by GSC)have reported 403 crawl error on some of my pages. The pages are actually working fine when loaded and no visible issue at all. My web developer told me that some times error issues are reported on a working pages and there is nothing to worry about.
                    My question is, will the 403 error have bad consequences on my SEO/Page ranking etc. These are some of the pages that have been reported with 403 error but loading fine: https://www.medistaff24.co.uk/hourly-home-care-in-evesham/ https://www.medistaff24.co.uk/contact-us/

                    Product Support | | ghrisa65
                    0
                  • DanielDL

                    Unsolved No replies from help@moz.com - one of our IPs is blocked by Cloudflare so we cannot access Moz Community from there

                    error access denied help

                    Hi all,
                    I am a bit at my wits end trying to get some acknowledgement from MOZ. Have had no replies, no ticket auto-replies, no updates on any of the messages I have sent via the Moz Help Form on the website. Literally nothing. I wanted to avoid having to post publicly, but does anyone know how to raise a "technical problem" ticket with MOZ? help@moz.com never replies and the Help Form doesn't generate any kind of ticket. From our main office we get an "Access denied" Error (via Cloudflare) specifically for the Moz Community area. This happened to us in February of this year and has been happening again all through May. After testing ourselves with our IT, we determine that MOZ's Cloudflare account has incorrectly blocked the dedicated IP address specific to the internet connection at our head office. This means that none of our Moz User accounts can access anything related to the Community area in our account when working at the studio. We can only do so when working remotely (ie. some other IP address). This is incredibly frustrating, particularly as we've been on a proper paid MOZ account for many years. And I have sent numerous email requests, messages via the Form, etc., and have never heard back from anyone at all. The problem has been on-going for some time and I guess it is my fault because I tried to politely wait a fair amount of time between each follow-up. Only to realize that, actually, I don't think anyone is monitoring help@moz.com or even the Form submissions, or are even looking into the issue for me. Am hoping this message is seen by someone at Moz so they can let me know what is going on please? Guys..... c'mon.....

                    Product Support | | DanielDL
                    0
                  • Yenlo

                    Website can't be crawled

                    Hi there, One of our website can't be crawled. We did get the error emails from you (Moz) but we can't find the solution. Can you please help me? Thanks, Tamara

                    Product Support | | Yenlo
                    0
                  • Karen_Dauncey

                    Crawl test

                    I used to use the crawl test tool to crawl websites and it presented the information in a really useful hierarchy of pages. The new on-demand crawl test doesn't seem to do this. Is there another tool I should be using to get the data?

                    Product Support | | Karen_Dauncey
                    0
                  • Benjamien

                    Crawling issue

                    Hello,
                    I have added the campaign IJsfabriek Strombeek (ijsfabriekstrombeek.be) to my account. After the website had been crawled, it showed only 2 crawled pages, but this site has over 500 pages. It is divided into four versions: a Dutch, French, English and German version. I thought that could be the issue because I only filled in the root domain ijsfabriekstrombeek.be , so I created another campaign with the name ijsfabriekstrombeek with the url ijsfabriekstrombeek.be/nl . When MOZ crawled this one, I got the following remark:
                     **Moz was unable to crawl your site on Feb 21, 2018. **Your page redirects or links to a page that is outside of the scope of your campaign settings. Your campaign is limited to pages with ijsfabriekstrombeek.be/nl in the URL path, which prevents us from crawling through the redirect or the links on your page. To enable a full crawl of your site, you may need to create a new campaign with a broader scope, adjust your redirects, or add links to other pages that include ijsfabriekstrombeek.be/nl. Typically errors like this should be investigated and fixed by the site webmaster. I have checked the robots.txt and that is fine. There are also no robots meta tags in the code, so what can be the problem? I really need to see an overview of all the pages on the website, so I can use MOZ for the reason that I prescribed, being SEO improvement. Please come back to me soon. Is there a possibility that I can see someone sort out this issue through 'Join me'? Thanks

                    Product Support | | Benjamien
                    0
                  • KBC

                    No crawl data anymore

                    Using moz quite some time, but I don't have any crawl data anymore. What happened? (www.kbc.be)
                    http://analytics.moz.com/settings/campaign/517920.1128516

                    Product Support | | KBC
                    0
                  • potterharry

                    I have removed a subdomain from my main domain. We have stopped the subdomain completely. However the crawl still shows the error for that sub-domain. How to remove the same from crawl reports.

                    Earlier I had a forum as sub-domain and was mentioned in my main domain. However i have now discontinued the forum and have removed all the links and mention of the forum from my main domain. But the crawler still shows error for the sub-domain. How to make the crawler issues clean or delete the irrelevant crawl issues. I dont have the forum now and no links at the main site, bu still shows crawl errors for the forum which doesnt exist.

                    Product Support | | potterharry
                    0

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy

                  Looks like your connection to Moz was lost, please wait while we try to reconnect.