Welcome to the Q&A Forum

Hello. I'm wondering if there is a way to control the crawl rate of MOZ on our site. It is hosted on Shopify which does not allow any kind of control over the robots.txt file to add a rule like this:

User-Agent: rogerbot Crawl-Delay: 5

Due to this, we get a lot of 430 error codes -mainly on our products- and this certainly would prevent MOZ from getting the full picture of our shop.

Can we rely on MOZ's data when critical pages are not being crawled due to 430 errors? Is there any alternative to fix this? Thanks

Hello Dave. Thanks for your reply. We are aware this is not affecting us being temporary and exclusive to the MOZ bot so that's why we are worried about the data-set issues.

As I mentioned most of our excluded content are products, we can't be certain that MOZ has every keyword and that the ones discovered are being weighted correctly.

Understandably Shopify might never make robots.txt available so it would be nice for MOZ to identify the web as a shop hosted on Shopify (a moz.txt file) and apply a rate limiting, at the very least allow the user to control the crawl parameters from our control panels for those SaaS apps that block these core functions.

Hope MOZ and Shopify one day have a coffee and find a way to figure this out. But meanwhile, Is there any way to request crawls in specific folders?  something like "domain.com/products/*****"