I’d like to set up a Moz campaign that crawls just the primary website, not subdomains
-
Is this something you could help with? (it either bombs or crawls everything, so I assume I’m missing something in the campaign settings, or it’s just not possible.
-
Hey, thanks for reaching out to us!
You can block our crawler from crawling specific subdomains of your site by excluding our user-agent from the subdomains robots.txt file. Our user agent is "rogerbot" and the directive would look something like this:
User-agent: rogerbot
Disallow: /You will want to make sure that this is associated with the robots.txt file specific to the subdomain you want to block.
You can read more about Rogerbot in our guide.
Hope that helps, let me know if you have further questions.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best Ways to Use Moz to Increase Rankings
Hello, I work for an online retailer and I have been using Moz for a few weeks now but have had limited success so I wanted to ask how I could better spend my time on this platform. I have been focusing heavily on the Page Optimization tool which has allowed me to rank high on the Google Shopping's Free Listings but it seems to have had little to no impact on the Google All search itself over those weeks. Perhaps I'm not utilizing it properly? I tend to focus on relevant keywords with high volume as identified by the Keyword Explorer tool. Alternatively, is there a Moz tool that might be more helpful? I can provide additional details or specific examples if needed. Thank you for your consideration,
Getting Started | | ForestGT
Forest1 -
How long does it take, So i can see results of Kewords Ranking search on moz?
Hi, I m the new guy at the Neighborhood Can you help please with the question... how long does it take, So i can see results of Kewords Ranking search on moz? This is the popup note Message: Moz collects and updates campaign data weekly. Not collected means we have not yet updated data for this keyword. TNX
Getting Started | | YarivHPAZ1 -
Does anyone know where I can find the Moz Video explaining how to use the Craw Diagnostic Feature? Thank!
I am starting to use the craw diagnostics(specifically duplicate content) and I know there was a very helpful tutorial video i saw earlier but I cant seem to find it now
Getting Started | | John-Francis0 -
Does the Moz crawler have a static ip address?
We block AWS from crawling our site. Does the moz crawler use a static ip address that we could allow? I'm currently not able to add a campaign because Moz can't connect to our site.
Getting Started | | uShip0 -
Pro just NOTworking, won't add campaigns...help!
I have been a member before, but cancelled due to Moz not doing daily ranking. Signed up again, as i like the reporting facility, and now have planned to use this along with Rankinity. I got the free trial, added in two new campaigns, which is not a quick process if you do it correctly, and neither are showing up in my campaigns tab. So I thought, maybe its because it's a free trial...so I upgraded to the 149/month, and got billed. Tried adding a campaign again, and nada. Also tried activating archived campaigns...again, nothing. Nothing is working, talking directly to Moz is pretty much impossible (no phone number, the 'talk to us' button never gets a response - tried that on Friday), so I am trying to get some kind of answers here. What I would like from Moz is: 1. Make the adding campaigns thing work asap (I need to make the client reports soon). 2. Make it so my old archived ones, when I click the activate button work. 3. If they are feeling nice for wasting hours of my time, maybe refunding my sub for the month and putting me back on the free trial or something. My head is about to explode....HELPPPPP! Love, Jeannie xo
Getting Started | | Jeannie.0 -
Crawl Diagnostics Help
Hi there Where can i find my campaigns crawl diagnostics? I need to find where this information can be found and specific issues? Is this possible, i cant seem to find this info. regards Ana
Getting Started | | Starsia200000 -
Why can't I add more than one campaign in my trial version of MOZ?
Hi, I'm on the 30 day trial and I can only run one campaign right now. I thought I could add in 5 campaigns. When I go into Manage Campaigns the 'add a campaign' button is light blue and I can't click it. Is this just because I'm on the trial? Or should I be able to add 4 more campaigns?
Getting Started | | Sophie-Kool0