Killing 404 errors on our site in Google's index

[seanmccauley]

Having moved a site across to Magento, obviously re-directs were a large part of that, ensuring all the old products and categories linked up correctly with the new site structure.

However, we came up against an issue where we needed to add, delete, then re-add products. This, coupled with a misunderstanding of the csv upload processing, meant that although the old urls redirected, some of the new Magento urls changed and then didn't redirect:

For Example:

mysite/product

would get deleted re-added and become:

mysite/product-1324

We now know what we did wrong to ensure it doesn't continue to happen if we weret o delete and re-add a product, but Google contains all these old URLs in its index which has caused people to search for products on Google, click through, then land on the 404 page - far from ideal.

We kind of assumed, with continual updating of sitemaps and time, that Google would realise and update the URL accordingly. But this hasn't happened - we are still getting plenty of 404 errors on certain product searches (These aren't appearing in SEOmoz, there are no links to the old URL on the site, only Google, as the index contains the old URL).

Aside from going through and finding the products affected (no easy task), and setting up redirects for each one, is there any way we can tell Google 'These URLs are no longer a thing, forget them and move on, let's make a fresh start and Happy New Year'?

[Marcus_Miller]

No canonical back to the main product page?

[seanmccauley]

Both helpful replies thanks. Further investigation led me to this Magento Bug:

http://www.magentocommerce.com/bug-tracking/issue/?issue=13662

(Need to have a magneto account to see the bug report).

Seems there's a spearate underlying issue which we need to fix first - the rewrite table grows exponentially every time we index Magento and creates a new URL for every configurable product. i.e. a product that has one or more associated products that will have the same name - used for displaying different sizes and colours. This means that Google is picking up a new page for each configurable product each time it indexes: different URL, same content, same product sku - a technical SEO nightmare!

[Marcus_Miller]

Hey Sean

This should take care of itself but there are a few things you can do to help.

**1. **Firstly, using webbug or some such, just make sure the page is returning a HTTP 404 or 410 code to ensure that whilst it may be displaying some kind of 404 like page, that it is actually sending the 4XX code back to Google (so they can update this and remove them).

2. Then, you can log into webmaster tools and remove URLs from your site:

Webmaster Tools > Optimisation > Remove URLs

This way you can manually remove them.

---

Alternatively, you could always just manually add some 301 redirects for those pages which may be the quickest way to sort this out and certainly provides the best experience for any users clicking on those links in the SERPs.

Hope that helps!
Marcus

[SoftzSolutions]

complex thing. Not sure if this may help you or not -

Example meta tag

Add the following meta tag in the HTML source of your page:

<meta http-equiv="expires" content="mon, 27 sep 2010 14:30:00 GMT">