Crawler triggering Spam Throttle and creating 4xx errors

[kbates]

Hey Folks,

We have a client with an experience I want to ask about.

The Moz crawler is showing 4xx errors. These are happening because the crawler is triggering my client's spam throttling. They could increase from 240 to 480 page loads per minute but this could open the door for spam as well.

Any thoughts on how to proceed?

Thanks! Kirk

[kbates]

Thank you Dave!

[dave.kudera]

Hey Kirk!  We built our crawler to obey robots.txt crawl-delay directives. In the future, if this is ever an issue, you can use the crawl delay to slow Rogerbot down to a more reasonable speed. However, we don't recommend adding a crawl delay larger than 10 or Rogerbot might not be able to finish the crawl of your site.

Just add a crawl delay directive to your robots.txt file like this:

User-agent: rogerbot
Crawl-delay: 10

Here's a good article that explains more about this technique: https://moz.com/learn/seo/robotstxt.  I hope this helps, feel free to reach out if you have any other questions!