When I crawl my site On Moz it says it can't access the robots.txt file, but crawl is fine on SEM Rush - Anyone know any reason for this?

[Webreviewadmin]

Hi guys,

When I try to run a site crawl on Moz it returns an error saying that it has failed due to an error with the robots.txt file.

However, my site can be crawled by SEM Rush with no mention of problems with roots.txt file issues.

My developer has looked into it and insists their is no problem with my robots.txt and I've tried the Moz crawl at least 6 times over an 8 week period.

Has anyone ever seen such a large discrepancy between Moz and SEM Rush or have any ideas why Moz has this issue with my site??

TIA everyone

[meghanpahinui]

Hi there!

Thanks so much for the great question! I'm so sorry you're having trouble getting your site crawled. Without knowing the exact site you're working with I can't say for sure what's going on but I can offer some suggestions that may help. If you're seeing the crawl is successful with other tools, there may be a bot-specific setting or directive that is banning our crawler, rogerbot, from your site. This may be coming directly from your server or it may be listed in your robots.txt file.

I have a troubleshooting guide here that may help- it outlines some other common issues that can keep rogerbot from being able to move forward with the crawl.

If you're still having trouble, please feel free to send an email on over to help@moz.com with the site you're having trouble with. That way we can take a look and see what's going on!

[adrian4131]

I also had this issue today on my site. Haven’t tried Semrush but got the same error that you are describing when I used Moz