Welcome to the Q&A Forum

Thank you. I'll try to solve the problem

The logs are like this:

"GET / HTTP/1.0" 400 166 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr2-crawler-101@moz.com)" "-" - "https"

and of course sometimes rogerbot is trying to see the robots file:

"GET /robots.txt HTTP/1.1" 400 166 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr2-crawler-101@moz.com)" "-" - "https"

for me it looks like the rogerbot is disallowed in robots.txt but the file is like this https://unplag.com/robots.txt

thanks a lot!

Thanks.

The last crawl was after the robots.txt change.

And I don't see any errors in the dashboard.

Thanks! I'll do that.

Hello

Please help me to solve the problem.

The on-page grader and Crawl Test are not working for Unplag.com website. Both said that they can't access the url. Yes, I've tried different variants like unplag.com, http://unplag.com

One more thing - RogerBot was disallowed in robots.txt file. I deleted it from the file a week ago so maybe moz index haven't been renewed.

Hello everyone.

I have the same issue with unplag.com. Both on-page grader and website crawl are not working.

Yes, I have tried different urls.

According to Charles Mazzie - moz index changes once a month so the problem can occur because a week ago I had Rogerbot disallowed in robots.txt.

Also, moz guide says that sometimes cookie policy of my site may cause RogerBot buzzing.

Unfortunately I still haven't solved the problem. So if you can - please help.