How to block Rogerbot From Crawling UTM URLs
-
I am trying to block roger from crawling some UTM urls we have created, but having no luck. My robots.txt file looks like:
User-agent: rogerbot Disallow: /?utm_source* This does not seem to be working. Any ideas?
-
Shoot! There may be something else going on. Give us a shout at help@moz.com and we'll see if we can figure it out!
-
FYI - I tried this and it did not work. Rogerbot is still picking up URL's we don't need. It's making my crawl report a mess!
-
The only difference there is the * wildchar. The string with that character will limit the crawler from accessing any URL with that string of characters in it.
-
What is the difference between Disallow: /*?utm_ and Disallow: /?utm_ ?
-
Hi there! Tawny from the Customer Support team here!
You should be able to add a disallow directive for that parameter and any others to block our crawler from accessing them. It would look something like this:
User-agent: Rogerbot
Disallow: ?utmetc., until you have blocked all of the parameters that may be causing these duplicate content errors. It looks like the _source* might be what's giving our tools some trouble. It looks like Logan Ray has made an excellent suggestion - give that formatting a try and see if it helps!
You can also use the wild card user-agent * in order to block all crawlers from those pages, if you prefer. Here is a great resource about the robots.txt file that might be helpful: https://moz.com/learn/seo/robotstxt We always recommend checking your robots.txt file with a handy Robots Checker Tool once you make changes to avoid any nasty surprises.
-
Skyler,
You're close, give this a shot:
Disallow: /*?utm_
This will be inclusive of all UTM tags regardless of what comes before the tag or what element you have first.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Crawling only the Home of my website
Hello,
Product Support | | Azurius
I don't understand why MOZ crawl only the homepage of our webiste https://www.modelos-de-curriculum.com We add the website correctly, and we asked for crawling all the pages. But the tool find only the homepage. Why? We are testing the tool before to suscribe. But we need to be sure that the tool is working for our website. If you can please help us.0 -
Solved Site Crawl Won't Complete
How can I start/restart a new site crawl? I requested one 2 days ago on one of my sites, and it won't complete. It's only 150 pages -
Product Support | | PaulBarrs0 -
Solved Why is MOZ crawl taking so long?
I began my site crawl on November 3rd and now it is November 7th and it is still "in progress". Why is this happening?
Product Support | | CarisaS_Wenda0 -
My site crawl has been in progress since last week
Hi there, I've been waiting on my site crawl to complete since Friday (it's Tuesday now), but it still has the 'in progress' notification at the top. Is it normal for it to take over 3 days? Or is there something holding it up?
Product Support | | VAPartners0 -
Which URL should I add at general settings?
Hi,I would like to double check something. We have set up our site to be international with multiple language. However we have at the moment only 1 language live. SO when you go to www.yourzite.com, you will be redirected to www.yourzite.com/nl/.Which URL should I add at the general campaign settings. I have now added www.yourzite.com/nl/, but it shows that I don't have any incoming links or domain authority. Should I change the link to yourzite.com? That is also the link that Google shows in the search results. Regards Jack
Product Support | | YourZite.com0 -
SEO Moz PRO app Isn't Crawling Anymore
Hi, We find the SEO Moz PRO app a great tool for us. What is the reason that it is not re-crawling the websites included in our campaigns anymore?
Product Support | | solution.advisor0 -
Latest Crawl Stats Not Showing
I got the email that Moz crawled my site for the week of February 8. I expected to log in and find some shinny new data, but new stats are not showing in my dashboard yet. Is there a delay? Is there something wrong?
Product Support | | ChadC0 -
Moz crawls
Hi all! I'm running two separate campaign crawls at the moment that I'm having a few issues with. The first is http://www.muchbetteradventures.com/. I'm tracking this at root to try an pick any problems with a previous version of the site, which is on the v1.muchbetteradventures subdomain. However, I'm only seeing 100 or so pages as being crawled in Moz, compared to thousands in Google. There's no access blocked alerts in Moz either. The second is http://www.sothebys.com/. I started this crawl at root a few days ago and no pages at all have been processed. Very odd! Any advice would be much appreciated.
Product Support | | neooptic0