Cleaning up 404s

[MarkWill]

I have recently moved to WordPress (from DNN) and I am very pleased with the change. As part of this switch I took the opportunity to totally revamp my posts, in terms of their URLs, titles, content, etc. I also put 301 redirects in place from the old to the new URLs.

Despite the redirects, logs suggest I have tons of 404s to my site. My site has been around for 20 years this year so I am not particularly surprised by this. Many of these broken URLs are one-offs (and I have no idea why they come in :)) and I am not worried about them. But over the years I have apparently had various versions of the same content, with different types of URL structure. By way of example, if I look at .aspx requests with "human readable" URL (where the URL is obviously an article/post) I have almost 2,500 such requests generating 404s in the last week or so.

My question is how much effort should I put into creating 301 redirects for these old ULRs? I see the recent changes as something of a fresh start so part of me wants to just ignore them and let search engines figure out the new URLs naturally (which seems to be happening). But I also wonder about how much SEO magic I am throwing away here - is it worth spending time reviewing these old URLs, identifying the new URL and implementing a redirect? That's a very time consuming process but I can make it happen if there's real value in doing so.

On a related question, once a search engine detects a 404 is it "done" with that 404 for good? For example, if Google tries to access a URL and finds a 404, does it just wipe the URL and never try again? If so, that presumably means - for that search engine - there's no point in the redirect being in place. It's too late at that stage, right?

Thanks.

Mark

[MarkWill]

Thank you, Dirk and Chris. Your comments are very much appreciated.

What I decided to do was basically redirect conditionally. What I mean by that is I did the following:

• Exported the 18,000+ 404s over the last week or so
• Created an Excel pivot table to count each URL
• Ordered by number of hits for each URL
• Ignored all URLs that have generated 10 or less 404s in the last week (this gets rid of thousands)
• Filtered out stuff that related to my old CMS (DNN) - these are clearly not relevant anymore. I could do this by pattern matching a few strings (/portals, etc) for DNN-specific URLs.
• Of the remainder, manually scanned for ad-hoc stuff that I know I won't need (old content, images, etc)

That little process got me down from 6,200+ URLs (accounting for the 18,000+ 404s) to a little over 200 URLs for which I will implement redirects. I'm starting at the top of the list (URLs that have generated the most number of 404s in the last week). Some have hundreds of 404s associated with them and there's a clear target URL, so it's good to do these.

I will work through these manually (takes a while :)) but at the end of this I think I will have covered the most important 404s. I'd fully expect my 404s over the next week to be much lower

Thanks again.

Mark

[GPainter]

Hello,

One thing you can try is crawling your site with screaming frog and get a list of all the urls, with said exported 404 you can check in most link checkers (e.g. OSE, Majestic, Ahrefs) via bulk checkers,with this you can roughly work out the highest value of those links as a way to prioritize your work, not the best solution but it may at least give you some where to start.

Regards to 404's Google does check them think of it like this, its looking through a site it will follow a link and if that links results in a 404 it would be marked down but next time the crawler comes to the site it will do the same thing lets say you've fixed that 404 it would then crawl that same page and find the new 301'ed content of however you fix it etc.

finally don't look at the pages as 404 look at them as lost content and add a value to that, if you have a missing page that in fact is a gem of content that was once useful and people are still clocking on that link looking for the same content then its defiantly worth trying to fix that, however pages with little content or value may not be as helpful it is always nice to have a "clean" site with 0 404s If you decide to not fix the 404's its not the worst thing in the world it's just a possible leak in link juice but if you're not fussed don't sweat it, especially if you can put the time from fixing thousands of 404's into creating awesome content etc.

Hope some of that helps, and good luck and happy 20th Birthday to your site!

[DirkC]

Hi

According to Matt Cutts a 404 means page is gone (but not necessarily permantent) - permanent would be 410 (although he also indicates that there is very little difference between these two from SEO perspective.

What to do with these 404 depends a bit on the situation

• if these pages have external links pointing to them - I would try to redirect (even better ask the one who's linking to update the links although maybe difficult in practice)

• if these are old url's which haven't been used for a while & don't generate traffic - just leave them - they will disappear

• where do these 404 come from - if they are just listed in WMT - you can ignore them. If it's actual people trying to visit your site on these pages I would try to redirect them to the appropriate new page (if not for the SEO than for the user experience)

• check that no internal links exist to these old 404 pages (Screaming Frog is made for this)

You say you 301'd the pages of the old site - did you check your landing page report in Analytics before migration - you should make sure that these top 5000 url's are properly redirected - normally Google will figure it out after a while, but it can have a negative impact on your results if a lot of these landingpages generate a 404.

As additional resource  - probably a bit too late now you could check the different steps in this guide: http://moz.com/blog/web-site-migration-guide-tips-for-seos to be sure that you didn't miss something important.

Hope this helps

Dirk