What to do about old urls that don't logically 301 redirect to current site?

[jcgoodrich]

Mozzers,

I have changed my site url structure several times.

As a result, I now have a lot of old URLs that don't really logically redirect to anything in the current site.

I started out 404-ing them, but it seemed like Google was penalizing my crawl rate AND it wasn't removing them from the index after being crawled several times. There are way too many (>100k) to use the URL removal tool even at a directory level.

So instead I took some advice and changed them to 200, but with a "noindex" meta tag and set them to not render any content. I get less errors but I now have a lot of pages that do this.

Should I (a) just 404 them and wait for Google to remove (b) keep the 200, noindex or (c) are there other things I can do? 410 maybe?

Thanks!

[irvingw]

"So instead I took some advice and changed them to 200, but with a "noindex" meta tag and set them to not render any content. I get less errors but I now have a lot of pages that do this."

I would not recommend keeping it that way. You could mass redirect them to the sitemap page if they are passing PR and or some traffic, and there is no logical other place to point them.

404's are not really something that can hurt you, providing that they are coming from external sources and you aren't providing 404 links on your site to dead pages on your site, if there are these, then you should fix the internal links at the source.

[David-Kley]

I dont think 404 errors hurt your site. If you have that many pages, they are most likely crawling your site a lot anyway. Have you set your crawl frequency in your sitemap? On bigger sites that get frequent updates, we set the crawl frequency to daily rather than weekly.

If possible, try to see if there are any top level items you can submit a URL removal request for. Hopefully this can speed up the process fo getting the URL's removed. This process can take a long time for Google to take care of. After changing websites we still had 404 errors after 6 months, even after submitting the URL removal request.

Another option is to have the page render a 410 rather than a 404. A 410 states to the search engine the page is gone, and will not be coming back. If you are using some form of cart system or cms there might be a way to apply the code to a large number of pages at once, rather than trying to manually code 100k pages.

"410 Gone
The requested resource is no longer available at the server and no forwarding address is known. This condition is expected to be considered permanent. Clients with link editing capabilities SHOULD delete references to the Request-URI after user approval. If the server does not know–or has no facility to determine–whether or not the condition is permanent, the status code 404 (Not Found) should be used instead of 410 (Gone).  This response is cacheable unless indicated otherwise."

Worse case scenero, you could set them to no-index, or just leave them be. Even if they dont lead anywhere logically, they could still bring you traffic. Or redirect them to the closest thing that is on the site currently.

[RobertFisher]

JC,

When you say ...started out 404-ing them...seemed like Google was penalizing my crawl rate..... etc. I have not seen where Google even algorithmically had any real issues with 404's. I your site has 500K pages and 100K are 404'd I do not think it would be a problem for Google per se. (You might have a searcher problem if these were pages that were bookmarked, lots of links, etc.) My caution would be that if you have a lot of pages on the site with links that still go to the 404 pages you could run into UX issues. 
For me, I would go with the 404's. I think they will get removed over time.

Best

[Gerald-Vandeveer]

When necessary, redirect relevant pages to closely related URLs. Category pages are better than a general homepage.

If the page is no longer relevant, receives little traffic, and a better page does not exist, it’s often perfectly okay to serve a 404 or 410 status codes.

[GPainter]

You could redirect them to something even remotely relevant even if its the homepage at the end of the day. What ever you do it going to take time and it's going to give you some sort of headache.

What would best suit a user who might land on an old link or somehow get to the page?  That would be the best way to find a solution. A good soft 404 or redirect tends to help here.

Best of luck though.