Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Tags
  3. rogerbot
Log in to post
  • awilliams_kingston


    rogerbot reporting

    I feel like I spend a lot of time setting false positives in my reports to ignore. Can I prevent Rogerbot from crawling pages I don't care about with robots.txt directives? For example., I have some page types with meta noindex and it reports these to me. Theoretically, I can block Rogerbot from these with a robots,txt directive and not have to deal with false positives.

    Reporting & Analytics | | awilliams_kingston
    0
  • paulavervo

    Unsolved

    rogerbot moz crawler

    The Moz.com bot is overloading my server. How to solve it?

    Moz Pro | | paulavervo
    0
  • awilliams_kingston

    Unsolved

    rogerbot 503 ideas

    A lot of my Moz duties seem to be setting hundreds of issues to ignore because my site was getting crawled while under maintenance. Why can't Rogerbot take a break after running into a few of these and then try again later? Is there an official code for Temporary Service Unavailability that can smart bots pause crawls so that they are not wasting compute, bandwidth, crawl budget and my time?

    Product Support | | awilliams_kingston
    0
  • awilliams_kingston


    crawler cloudflare rogerbot 520 server error

    I am getting a lot of 520 Server Error in crawl reports. I see this is related to Cloudflare. We know 520 is Cloudflare so maybe the Moz team can change this from "unknown" to "Cloudflare 520". Perhaps the Moz team can update the "how to fix" section in the reporting, if they have some possible suggestions on how to avoid seeing these in the report of if there is a real issue that needs to be addressed. At this point I don't know. There must be a solution that Moz can provide like a setting in Cloudflare that will permit the Rogerbot if Cloudflare is blocking it because it does not like its behavior or something. It could be that Rogerbot is crawling my site on a bad day or at a time when we were deploying a massive site change. If I know when my site will be down can I pause Rogerbot? I found this https://developers.cloudflare.com/support/troubleshooting/general-troubleshooting/troubleshooting-crawl-errors/

    Technical SEO | | awilliams_kingston
    0

Looks like your connection to Moz was lost, please wait while we try to reconnect.