How to turn off automated site crawls

[SEONOW123]

Hi there,

Is there a way to turn off the automated site crawl feature for an individual campaign?

Thanks

[A Former User]

No problem!

Thanks for the suggestion!

I will pass that on as feedback

Eli

[SEONOW123]

Hi Eli,

Thanks for your response.

I'd really suggest that the functionality to prevent the site crawl feature is built into the campaign tool. It would take dev effort to disalow the Moz bot where I'm based.

Thanks

[A Former User]

Hey!

Thanks for reaching out to us!

Moz uses a crawler called Rogerbot to crawl your site and populate the "Site Crawl" section of your Campaign. Roger isn't a search engine index, so if URLs aren't blocked by the robots.txt file, then Roger will crawl them!

If you'd like to remove pages from the crawl, you can disallow them in the robots.txt file. Here's what that exclusion would look like if you were wanting to keep him from crawling any pages on your site:

User-agent: rogerbot
Disallow: *

I'd recommend checking your robots.txt file in this handy Robots Checker Tool once you make changes to avoid any nasty surprises.