Why are Pages returning 404 errors not being dropped?

[Towelsrus]

Our webmaster tools continues to return anywhere upwards of 750 pages that have 404 errors. These are from pages of a previous site no longer used.

However this was over 1 year ago these pages were dropped along with the 301 re-directs. Why is Google not clearing these from webmaster tools but re-listing them again after 3 month cycle? Is it because external sites have links to these pages?

If so should I put a 301 in place (most of these site are forums and potentially dodgy directories etc from previous poor link building programs) or ask for a manual removal?

[Towelsrus]

Thanks tom for all your help.

Regards

Craig

[TomRayner]

Very good point you've raised - 301ing those URLs effectively makes the links to your site "live" again.  If the links sit on a dodgy/spammy/poor quality page, then it could harm your site and I wouldn't put the redirect in place.

By in large, if you're beginning to doubt whether the link is worthwhile or not, chances are its not.  So if you have a bit of doubt about the link, then don't put the 301 in place.

[Towelsrus]

Hi Tom,

That more than explains it and gives me the answers. If I put 301 redirects in place what will happen if any of these external links are bad, will it harm our site? Its taken me many months to deal with duplicate content issues, canonicalisation of the site and much more. It was a complete mess and I don;t want to harm any good that come of all this.

[TomRayner]

Hi Craig

You touched on one of the reasons this is happening in your post - you could external links to these pages.  Also, they could still be appearing in the sitemap.

If you go into Webmaster tools > Health > Crawl Errors > Not Found and then click on one of the URLs, you can check whether or not the page is in the sitemap or whether it is being linked to from somewhere.

If you have external links, you have four options.  First, you could attempt to change the URLs on the pages they're being linked from.  This could be difficult and/or long.  Second, as you say, you could 301 redirect.  This would be useful if people are coming through those sites still, as you'll be fixing their user journey.  It would also pass on any link "juice" that page has to another.  Third would be to start returning a 410 error.  This explains 410 response codes - it basically tells the Googlebot to treat the URL as gone permanently.  This can be a bit tricky to setup and you have to be sure you want use the URL again in the future.

Finally, you could leave the 404s in place.  If none of the pages have any strength, no referral traffic is coming from them and they aren't interrupting a user journey in any way, I would simply leave them.  Google knows that 404s are just a matter of process and so recognises that 404 errors are simply a natural occurrence.  It would only ever be a problem if you returned tens of thousands of them, so you may just want to leave them be.

I would probably 301 redirect any old pages carrying strength to relevant equivalents (if not, the root domain) and leave the other 404s in place.  I would rewrite ASAP any URL that is interrupting a user journey.

Hope this helps!