Filtered Navigation, Duplicate content issue on an Ecommerce Website

[13375auc3]

I have navigation that allows for multiple levels of filtering. What is the best way to prevent the search engine from seeing this duplicate content? Is it a big deal nowadays? I've read many articles and I'm not entirely clear on the solution.

For example.

You have a page that lists 12 products out of 100:

companyname.com/productcategory/page1.htm

And then you filter these products:

companyname.com/productcategory/filters/page1.htm

The filtered page may or may not contain items from the original page, but does contain items that are in the unfiltered navigation pages. How do you help the search engine determine where it should crawl and index the page that contains these products?

I can't use rel=canonical, because the exact set of products on the filtered page may not be on any other unfiltered pages. What about robots.txt to block all the filtered pages? Will that also stop pagerank from flowing? What about the meta noindex tag on the filitered pages?

I have also considered removing filters entirely, but I'm not sure if sacrificing usability is worth it in order to remove duplicate content. I've read a bunch of blogs and articles, seen the whiteboard special on faceted navigation, but I'm still not clear on how to deal with this issue.

[caseyhen]

Hi Dstrunin,

I would still use the rel canonical tag even with or without the filter in place.  So if you have a list of products displayed unfilter at companyname.com/productcategory/page1.htm, I would add a rel canonical  with it pointing at companyname.com/productcategory/page1.htm.  For the filtered results,companyname.com/productcategory/filters/page1.htm , the canoncial tag would still point to companyname.com/productcategory/page1.htm.

It doesn't hurt to have a canonical tag point to the same page it's on.

If you can't do that I would meta noindex those filtered pages and remove the robots.txt stuff.  Robots.txt doesn't tell Google they can't index it it only says they can't crawl it.  So they could still index old stuff they crawled before you did the robots.txt stuff or index the title tags.

Casey

[13375auc3]

I have been doing that, but robots.txt only does so much. I've implemented the meta noindex tag as well and it doesn't seem to be taking all the pages out of the index.

[Goetzman]

My unprofessional opinion would be to use robot.txt on some areas. I'll also be interested to see what the pros here say.