Skip to content

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  • Unsolved

    crawl error crawling crawl

    I don't understand why MOZ crawl only the homepage of our webiste We add the website correctly, and we asked for crawling all the pages. But the tool find only the homepage. Why? We are testing the tool before to suscribe. But we need to be sure that the tool is working for our website. If you can please help us.

    Product Support | | Azurius
  • Unsolved

    crawl error

    Hi Moz crawler keep failing on my site with the error showing : Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. I'm not sure what am I missing out.. this is my robots.txt.. i don't think Im missing anything else.. can the support team help ?

    Moz Pro | | teikh
  • Unsolved

    crawl error robots.txt

    Hi all, im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. This is my robots.txt file : I'm not sure what else am I missing.. can anyone help

    Product Support | | teikh
  • Unsolved

    crawl error url issue

    I recently recieved a slew of content crawl issues via Moz for URL's that I have never seen before For example:
    Standard URL:,
    Newly identified URL: Does anyone know where the URL comes from and how to fix it?

    Moz Pro | | HannahPalamara
  • Unsolved

    roger-bot crawl error

    Hi, We're trying to get MOZ to crawl our site, but when we Create Your Campaign we get the error:
    Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct. If the issue persists, check out this article for further help. robot.txt is fine and we actually see cloudflare is blocking it with block fight mode. We've added in some rules to allow rogerbot but these seem to be getting ignored. If we use a robot.txt test tool ( with rogerbot as the user agent this get through fine and we can see our rule has allowed it. When viewing the cloudflare activity log (attached) it seems the Create Your Campaign is trying to crawl the site with the user agent as simply set as rogerbot 1.2 but the robot.txt testing tool uses the full user agent string rogerbot/1.0 (, albeit it's version 1.0. So seems as if cloudflare doesn't like the simple user agent. So is it correct the when MOZ is trying to crawl the site it uses the simple string of just rogerbot 1.2 now ? Thanks
    Ben Cloudflare activity log, showing differences in user agent strings

    Moz Pro | | BB_NPG

  • 403 errors crawl error

    My wordpress website has 162 crawl 403 errors. Based on what I read it means that the server is denying crawlers to access the pages. The pages itself will load so guessing it's just an issue with crawlers only. How do I go about fixing this issue?

    On-Page Optimization | | emrekeserr3
  • Unsolved

    crawl error crawl errors

    Moz is being blocked from crawling the following site - When looking at Robot.txt, the following is disallowing access but don't know whether this is preventing Moz from crawling too? User-agent: *
    Disallow: /adeci/
    Disallow: /core/
    Disallow: /connectors/
    Disallow: /assets/components/ Could something else be preventing the crawl?

    Moz Pro | | danhart2020
  • Unsolved

    crawl error crawl in progress crawl stalled

    The latest crawl on my site was the 4th Jan with a current crawl 'in progress'. How do i cancel this crawl and start a new one? I've been getting keyword ranking etc but no new issues are coming through. Screenshot 2022-05-31 083642.jpg

    Moz Tools | | ClaireU
  • Unsolved

    25 404 error crawl error

    Hi Community, has anyone else had a 404 error reported by Moz, where the end of the domain is /%25s? The error comes from my blog home page But when I look at the source code, I can't see anything that has a space at the end of the URL. I wonder if it is to do with the WordPress search? Thanks in advance for any insight.

    Moz Pro | | kaydeeweb
  • Unsolved

    performance metrics crawl error

    I am getting an error:
    Crawl Error for mobile & desktop page crawl - The page returned a 4xx; Lighthouse could not analyze this page.
    I have Lighthouse whitelisted, is there any other site I need to whitelist? Anything else I need to do in Cloudflare or Datadome to allow this tool to work?

    Product Support | | bhsiao 0

Looks like your connection to Moz was lost, please wait while we try to reconnect.