Intermittent 404 - What causes them and how to fix?

[daniel-brooks]

Hi!

I'm working on a client site at the moment and I've discovered a couple of pages that are 404ing but producing a 200 OK response.

However, I have checked these URLs again and some are now producing a 404 Error response. No changes have been made (that I'm aware of) so it appears that the URLs are returning both 200 OK and 404 Error responses intermittently.

Any ideas what could cause this and the best solution?

Thanks!

[seanginnaw]

Hi Daniel,

That does seem very odd!

There can be various different things at play here in my experience:

1. Publishing and un-publishing in the CMS - Sometimes clients and CMS users will switch the page publish status around which can make it difficult to keep track of whether the page exists or not
2. Removal of redirects - Much the same as above, there's the possibility that someone is adding/removing redirects. This could also be something to do with the removal of the server level redirects file, be that htaccess or IIS.
3. Crawling tool issue - If you're using a crawling tool (especially if doing so with JS rendering active), sometimes this can put a lot of pressure on the server and it can spit out the occasional error where there actually isn't one

It could also be a problem with how you're handing hard 404 errors vs soft 404s - i.e. actual not founds vs pages that don't function but the server is under the impression that they're fine.

Best of luck!

Sean