I’d like to set up a Moz campaign that crawls just the primary website, not subdomains
-
Is this something you could help with? (it either bombs or crawls everything, so I assume I’m missing something in the campaign settings, or it’s just not possible.
-
Hey, thanks for reaching out to us!
You can block our crawler from crawling specific subdomains of your site by excluding our user-agent from the subdomains robots.txt file. Our user agent is "rogerbot" and the directive would look something like this:
User-agent: rogerbot
Disallow: /You will want to make sure that this is associated with the robots.txt file specific to the subdomain you want to block.
You can read more about Rogerbot in our guide.
Hope that helps, let me know if you have further questions.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New to SEO? Check out Moz Academy!
Easy-to-access SEO education has always been dear to our mission. Simply log in with your Moz Profile and start learning. Whether you're new to SEO or looking to take your skills to the next level, Moz Academy has fun, affordable courses for the entire team! Check it out now! moz-academy-training.png
Getting Started | | HayleyBowyer2 -
Can someone help me to gain Moz trust and domain authority?
Hi someone help me how to gain moz trust and domain authority for my web site. here is my web addresss: www.bassinotary.com/ please tell me how i can improve ranking for my site. please help. thanks.
Getting Started | | grbassi0 -
Setting timescales in Custom Reports
Hi, is it possible to set a timescale throughout a Custom Report without having to remember to change the timescale for each individual module?
Getting Started | | mfrgolfgti0 -
My campaign hasn't updated since wk of Sept 18-24
Even though it shows it should run weekly the campaign appears to be stuck. Also, it seems like campaign frequency should be a variable under campaign settings, but isn't. Is there a way to manually run the campaign to add the data?
Getting Started | | dwerkema0 -
Why cant I add my site to a campaign?
I am trying to add my site www.dominickdalsanto.com to a campaign and it keeps telling me the URL is invalid. I have tried entering it as: dominickdalsanto.com dominickdalsanto.com/ www.dominickdalsanto.com http://www.dominickdalsanto.com Nothing works. I even tried the redirect domains I have ayres-seo.com and still nothing. I tried a few of my other sites too and it works for only one of them. It also would not take dominickinargentina.com Can someone help me please? Thanks!
Getting Started | | Ayres-SEO0 -
Why Moz New Users Are Left Alone?
New users when sign up to an application like Moz need concrete support to make them familiarized the basics at least, but Moz seems to be not realizing this hardship of the new users or intentionally ignoring it. There is no chat service or phone numbers to quickly contact the designated agent for some quick help. Mail responses that are received after much time have their limitations -- not much interactive. What the new users should do, because the Moz is not that simple for new users. [edited question for formatting]
Getting Started | | Sequelmed1 -
Why does Moz Analytics only let you look at last month / last week?
I cant think of any other reporting product that only lets you look at the last period, am I missing a setting somewhere. I need to take a look at Feb!
Getting Started | | locusm1 -
How to get moz to crawl a staging domain that is blocked by robots.txt
Is it possible to get Moz to do a crawl report on a domain blocked by robots.txt and actually display all errors instead of only one saying the domain was blocket in robots.txt? Anything i can add to robots.txt to make moz able to do the crawl report but still hinder google from crawling a staging domain?
Getting Started | | classifiedtech0