I’d like to set up a Moz campaign that crawls just the primary website, not subdomains
-
Is this something you could help with? (it either bombs or crawls everything, so I assume I’m missing something in the campaign settings, or it’s just not possible.
-
Hey, thanks for reaching out to us!
You can block our crawler from crawling specific subdomains of your site by excluding our user-agent from the subdomains robots.txt file. Our user agent is "rogerbot" and the directive would look something like this:
User-agent: rogerbot
Disallow: /You will want to make sure that this is associated with the robots.txt file specific to the subdomain you want to block.
You can read more about Rogerbot in our guide.
Hope that helps, let me know if you have further questions.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New to SEO? Check out Moz Academy!
Easy-to-access SEO education has always been dear to our mission. Simply log in with your Moz Profile and start learning. Whether you're new to SEO or looking to take your skills to the next level, Moz Academy has fun, affordable courses for the entire team! Check it out now! moz-academy-training.png
Getting Started | | HayleyBowyer2 -
Setting up Moz Pro campaign for a subfolder
Hi, I'm just setting up my first campaign and would like some advice please. I have a website and also within it a a number of pages in a sub folder. These are technically a different business to the main website and I would like to track them separately in Moz so they can have their own keyword lists and competitors etc. How do I do this? Do I set up two separate campaigns www.site.com
Getting Started | | craigramsay
www.site.com/subfolder I understand the first one might still report on the subfolder but I guess I could just ignore this. Thanks0 -
Why is Moz.com saying that none are linking to www.oneworldcetner.eu
It still says 0 after 6 weeks if i use other tooks like http://openlinkprofiler.org/r/oneworldcenter.eu#.VhbF7ivzJ8E I do see the backlinks that i was counting on was there
Getting Started | | onewordcenter0 -
Website errors?
Where can I see my domain website errors. Things like how may pages are missing meta description, duplicate title tags or broken links. I use to see it when I signed in. Now I can't find it.
Getting Started | | gsam1 -
Moz Analytics - How can I turn a whole report to 'Monthly'?
Hello, After spending 3 days setting up 14 clients on this system, by this, I mean I went through each client and re-made the same report 14 times as there is no alternative..I have noticed on my reports that: 'Dashboard' is set to 'Weekly' Social is set to 'Daily' Branding is set to 'Monthly' First questions, when I try to run a report, I go to Add Module and change 'Weekly' to 'Monthly', but when I press next, it changes itself back again, am I doing something wrong? UPDATE: I have found that I have go back in the reports, change the heading to 'Monthly' and then re-add this section again. How can I just run a whole report with monthly values? Surely, I do not have to re-do the entire report AGAIN, just to get a different value? I am mortified at how unfinished this product is, if this is the case.
Getting Started | | Paul_Tovey0 -
Fastest way to set up Moz to produce a list of easy-to-rank for keyword opps
I've used Hubspot's keyword tool previous to MOZ, and I know that they import data generated from MOZ into their tool... but I find it a much more convoluted process using Moz's more robust system. ALL I want is to dump in a long list of keyword possibilities, then see where we can easily rank for these terms, especially against our local competitors. How do I set this up???? It's not as self-explanitory as Hubspot! Thanks!!!
Getting Started | | uptongirl340 -
Cant download my crawl csv
When I click on the [download csv] in my crawl campaign section nothing happens.
Getting Started | | digitalmedialounge0 -
How do you "Moz Crawl" a website? Newbie...
Hi everyone; I've used Screaming Frog in the past and it's simple: you enter the URL to the box and click "start" and.. voila: as the button says, the crawling starts. I've had the Pro version of Moz for a while now and haven't really 'done' anything with it. I'd like to crawl a website and thought it would be as easy as it's always been with Screaming Frog... but, for some reason, I can't find the 'way' to do it. I find it really frustrating especially cause I feel like an idiot going around in circles thinking I'm missing something really obvious... Until I realised the only solution was to ask here! So... how in the world do you crawl a website using Moz tools? (Pro version) Thanks!
Getting Started | | patrihernandez1