Crawler triggering Spam Throttle and creating 4xx errors
-
Hey Folks,
We have a client with an experience I want to ask about.
The Moz crawler is showing 4xx errors. These are happening because the crawler is triggering my client's spam throttling. They could increase from 240 to 480 page loads per minute but this could open the door for spam as well.
Any thoughts on how to proceed?
Thanks! Kirk
-
Thank you Dave!
-
Hey Kirk! We built our crawler to obey robots.txt crawl-delay directives. In the future, if this is ever an issue, you can use the crawl delay to slow Rogerbot down to a more reasonable speed. However, we don't recommend adding a crawl delay larger than 10 or Rogerbot might not be able to finish the crawl of your site.
Just add a crawl delay directive to your robots.txt file like this:
User-agent: rogerbot
Crawl-delay: 10Here's a good article that explains more about this technique: https://moz.com/learn/seo/robotstxt. I hope this helps, feel free to reach out if you have any other questions!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
The factors considered in the new domain authority algorithm? On-site factors we can use to compare with competitors? Is having an NA as a spam score a bad thing?
Does anyone know the factors considered in the new domain authority algorithm other than spam score and complex distributions of links based on quality and traffic? Does anyone know of on-site factors we can use to compare with competitors to try and improve DA? Is having an NA as a spam score a bad thing?
Moz Bar | | CQMarketing0 -
Spam Score Help Needed!
Hi guys, Basically, long story short our site currently has a spam score of 5%, which is heading into dangerous territory. We have been looking into why this might be and the more we've looked into it the more stumped we've become. We compared our inbound links to our competitors, who's spam score currently stands at just 1%, and found that we actually had a smaller proportion of spammy inbound links compared to theirs - and yet our score is 5x higher. We have looked into disavowal, but this seems like a high-risk venture and something of a last resort so we'd rather see if there's anything else we can do about this before we press the button. Are there any other factors we need to look in to aside from inbound links and domains? Could it be something to do with our site architecture? For example we have a very high number of internal links (102,011, 94.1% are followed) compared to our competitors. This could qualify us as a 'large' site, and when paired with the fact that we only have 640 inbound links could be a big red flag to Google? Any help you'd be willing to provide us on this would be much appreciated!
Moz Bar | | rawdog0 -
Spam score of 28 %
Hello, I checked my website spam score in link explorer and found out it has got 28 %. What do you recommend I do ? should I ask for all the links to be removed (even though some of them are no followed) Is it possible that google has penalised me because of that score and de-ranked me ? Thank you,
Moz Bar | | seoanalytics1 -
Moz not updating the spam score metrics
Hi Experts, I have done everything to come over the following metrics flagged as spam on my website by Moz, few months ago. But, Moz has not updated the spam score yet. ✓Low MozTrust or MozRank Score - improved from 2 to 4.5
Moz Bar | | jamesh.rich01
✓Large Site with Few Links - My website have more than 6K backlinks
✓Small Proportion of Branded Links - My website have a good amount of branded backlinks
✓Thin Content - Every webpage on website has more than 500 words content
✓External Links in Navigation - There is no external link in navigation other than social media links
✓No Contact Info - The proper address has alreay been placed on website footer
✓Low Number of Pages Found - I am wondering if there are any standard score or number of links to reach to remove these flag?
Also, please suggest some ways that will help me improve moz spam score at faster rate. Thanks for your help in advance!"0 -
Crawl report shows that it gets 4xx errors for pages that work fine. Why?
On the crawl report it has all these "Critical Crawler Issues". They all say "4xx Error", yet when i click on the link from the crawler report, it goes to a perfectly functioning page, not a 404 page or anything. If i click in it actually says it's a 403 error. It's all for pages generated by the IDX solution for our real estate website. Is Moz broken or am i missing something? Here are a couple examples: <dl class="crawl-page-details-list"> <dd class="crawl-page-details-list-emphasis">https://teamvivi.com/homes-for-sale-map-search/</dd> <dd class="crawl-page-details-list-emphasis"> <dl class="crawl-page-details-list"> <dd class="crawl-page-details-list-emphasis">https://teamvivi.com/email-alerts/</dd> </dl> </dd> </dl>
Moz Bar | | TeamViviRealEstate0 -
All of a sudden a number of my key pages are getting 403 errors with Moz?
One of my squarespace sites has suddenly thrown up a number of 403 access denied errors for a range of pages on the site, according to the Moz weekly report. There is no access issues for these pages and nothing has been changes wrt url, etc..... so why the errors (which I am not seeing through SEM rush for example)? Thx, Phil
Moz Bar | | bugdoctor0 -
Problem Downloading Crawl Error Report PDF's
I am trying to download the PDF reports for the various 'crawl errors' - now some of them are quite large but would that justify why I am unable to download - the error is a straightforward one, see attached. Any ideas? Andy aDlViIN
Moz Bar | | TomKing0 -
WP 4.0 Update Causing Major Duplicate Content Errors?
According to my moz analytics, my site has went through the roof with duplicate content. There's a nice Mozzer named Abe looking into this with me, but I'm wondering if it could be due to the WP 4.0 update. Has anyone else experienced an uptick like this before? I've never had any problems with the other updates. Thanks, Ruben
Moz Bar | | KempRugeLawGroup0