Magento Hidden Products & Google Not Found Errors

[Marketing.SCG]

We recently moved our website over to the Magento eCommerce platform. Magento has functionality to make certain items not visible individually so you can, for example, take 6 products and turn it into 1 product where a customer can choose their options. You then hide all the individual products, leaving only that one product visible on the site and reducing duplicate content issues.

We did this. It works great and the individual products don't show up in our site map, which is what we'd like. However, Google Webmaster Tools has all of these individual product URLs in its Not Found Crawl Errors.

! For example:

White t-shirt URL: /white-t-shirt

Red t-shirt URL: /red-t-shirt

Blue t-shirt URL: /blue-t-shirt

All of those are not visible on the site and the URLs do not appear in our site map. But they are all showing up in Google Webmaster Tools.

Configurable t-shirt URL: /t-shirt

This product is the only one visible on the site, does appear on the site map, and shows up in Google Webmaster Tools as a valid URL. !

Do you know how it found the individual products if it isn't in the site map and they aren't visible on the website? And how important do you think it is that we fix all of these hundreds of Not Found errors to point to the single visible product on the site? I would think it is fairly important, but don't want to spend a week of man power on it if the returns would be minimal.

Thanks so much for any input!

[maartenvr]

I also have this same issue.... looking for a solution.

[carlymessmer]

Hi there,

I know this is an older question, but I wanted to follow up to see if you found a solution. We're currently experiencing the same issue with simple/not visible individually URLs appear in WMT's crawl error report.

Thanks in advance for any help!

[MikiP]

The problem I am currently facing is a site where the individual 'Simple Products' were not set to be invisible.  They have all been indexed!

So I have the same question really.

[Marketing.SCG]

Yes, the page is modified based on selected options and it is done via Javascript.

[SystemIDBarcodes]

OK. In a product page, the individual page is modified by a customer selecting options? If so, is that accomplished with Javascript substitution?

[Marketing.SCG]

So those individual pages never actually appear on the site. They are just created to allow Magento to pull inventory on those items from that configurable product, which is why I'm not sure how Google is finding them.

For example, if I go to mysite.com/white-t-shirt, I would get a 404 (and if I searched for it, nothing would come up) because as far as the world outside of Magento admin is concerned, that URL doesn't exist.

[SystemIDBarcodes]

How do those pages appear? In response to an AJAX call? A sitesearch? I've been hearing that Google has increased their ability to crawl pages that appear through dynamic calls.