Rogerbot does not catch all existing 4XX Errors

[inlinear]

Hi I experienced that Rogerbot after a new Crawl presents me new 4XX Errors, so why doesn't he tell me all at once?

I have a small static site and had 9 crawls ago 10 4XX Errors, so I tried to fix them all.
The next crawl Rogerbot fount still 5 Errors so I thought that I did not fix them all... but this happened now many times so that I checked before the latest crawl if I really fixed all the errors 101%.

Today, although I really corrected 5 Errors, Rogerbot digs out  2 "new" Errors. So does Rogerbot not catch all the errors that have been on my site many weeks before?

Pls see the screenshot how I was chasing the errors

404.png

[inlinear]

I understand,

I am not using a CMS and the site is not very big, so I wondered why Roberbot did not find all the 404 Error at the first time, because they have been there for many months.

Holger

[JoelDay]

Hey Holger,
Our crawler will catch as many errors as it can. It's possible that these errors were not present or just were not found at the time of the crawl.

I'm running a crawl test to see if there's any discrepancy between your current campaign crawl and mine just to double-check.

In general, Kyle is correct that sometimes those errors just crop up, especially if you're using any sort of CMS.

I hope that helps. I'll update here after my crawl test is done.
Cheers,
Joel.

[KyleJack]

Hi Holger,

4XX Errors can be quite common depending on your site setup so don't be surprised that Roger will keep returning errors for you to fix.

I would advise checking this data against GWT's own crawl error data which you can find in Webmaster Tools under Health>Crawl Errors.

I hope that helps,

K