Robots.txt & Duplicate Content

[Careerbags]

In reviewing my crawl results I have 5666 pages of duplicate content. I believe this is because many of the indexed pages are just different ways to get to the same content. There is one primary culprit. It's a series of URL's related to CatalogSearch - for example; http://www.careerbags.com/catalogsearch/result/index/?q=Mobile

I have 10074 of those links indexed according to my MOZ crawl. Of those 5349 are tagged as duplicate content. Another 4725 are not.

Here are some additional sample links:

http://www.careerbags.com/catalogsearch/result/index/?dir=desc&order=relevance&p=2&q=Amy
http://www.careerbags.com/catalogsearch/result/index/?color=28&q=bellemonde
http://www.careerbags.com/catalogsearch/result/index/?cat=9&color=241&dir=asc&order=relevance&q=baggallini

All of these links are just different ways of searching through our product catalog. My question is should we disallow  - catalogsearch   via the robots file? Are these links doing more harm than good?

[DonnaDuncan]

For product pages, I would canonical the page with the most descriptive URL.

For category pages, I agree with you, I would noindex them.

I think I just answered my own question!!

[WesleySmits]

Oke, the question concerning rel="canonical" is which URL becomes the canonical version? Since there is no page on the website which would be appropiate (as far as i've seen) i recommended the meta robots tag.

I do agree that rel="canonical" is the preferred option, but in this situation i can't see a way to implement it properly. Which page would you highlight as the canonical?

[DonnaDuncan]

I agree entirely that "Search result pages are too varied to be included in the index".

That said, my understanding is that if you canonical a page, it doesn't get indexed. So we wouldn't have to worry about the appearance / user-friendliness of the URL. But (again, in my opinion) we should still worry about link equity being passed, and that won't happen if you noindex.

This gets complicated fast. I like your solution b/c it's a lot cleaner and easier to implement. Still not convinced it's the "best" way to go though.

[Stewart_SEO]

Where is the evidence that these work? I have never seen them work. Google totally ignores the URL parameters tools in GWTs.

[WesleySmits]

I do agree that a rel="canonical" is good option for the problem that's at hand.
As jeremy has stated however the link we are referring to in the href section redirects to the home page. http://www.careerbags.com/catalogsearch/result/index/

In my original answer i did not test this. I assumed there would be a list of all products here not filtered by search results. Since this is not the case and this page in fact does not exist it's hard to point at a url to be canonical.

Therefor i changed my answer to include the robots meta tag. This would indeed remove the search pages from the search index. I do think this is a positive thing though.

Look at the following url: http://www.careerbags.com/catalogsearch/result/?q=rolling+laptop+bags

Not really the type of URL i would click on in the search results. The following URL however is something i would want to click on: http://www.careerbags.com/laptop-bags/women-s/rolling-laptop-bags.html

Search result pages are too varied to be included in the index to my opinion.

Hope you agree with this, if not then i would like to hear your thoughts on this.

[DonnaDuncan]

Simon, Wesley, Michael...

These customer facing search result pages are the ones often bookmarked and shared by site visitors. How worried does one need to be about losing link equity? I realize every site is going to be different and social shares don't have link equity - at least for now - but this could add up over time. The rel canonical will enable capture of link equity whereas the robots noindex will not.

Am I over thinking this?

[WesleySmits]

In this case you could add the meta robots tag on the search result pages like this:

content="noindex, follow">

Search results can indeed spawn an infinite amount of different URL's. This can be avoided by making sure they are not included in the index but are followed.

[simon_realbuzz]

Webmaster guidelines specifically request that you prevent crawling of search results pages using a robots.txt file. The relevant section reads: "Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines."

[MickEdwards]

There are 2 distinct possible issues here

1. Search results are creating duplicate content

2. Search results are creating lots of thin content

You want to give the user every possibility of finding your products, but you don't want those search results indexed because you should already have your source product page indexed and aiming to rank well.  If not see last paragraph.

I slightly misread your post and took the URLs to be purely filtered. You should add disallow  /catalogsearch to your robots.txt and if any are indexed you can remove the directory in Webmaster Tools > Google Index > Remove URLs > Reason: Remove Directory.  This from Google - http://www.mattcutts.com/blog/search-results-in-search-results/

If your site has any other parameters not in that directory you can add them in Webmaster Tools > Crawl > URL Parameters > Let Googlebot Decide. Google will understand they are not the main URLs and treat them accordingly.

As a side issue with your search results it would be a good idea to analyse them in Analytics. You might find you have a trend, maybe something searched for or not the perfect match for the returned result, where you can create new more targeted content.

[Careerbags]

I'm not sure this is the right approach. The catalog search is based on the search box on the website. The query parameter can be anything the customer enters. Are you suggesting that the backend code be modified to always return the  in every result?

And why that page because that URL just redirects to the home page because there is no query parameter provided for the search.

In terms o losing link equity, how much equity do they have it they are duplicate content?

[DonnaDuncan]

Hi Jeremy.

Yours is a common problem. The best way to deal with it is, as Wesley mentions, by putting canonical tags on all the duplicate pages - the one you want indexed and to show up in search results AND all the others that you can arrive at via catalog search or any other means of navigation.

Michael's suggestion will prevent the duplicate pages from getting indexed by Google. Unfortunately you lose any link equity going that route, so I'd suggest starting with canonical tags first.

[MickEdwards]

To back up the detail Wesley gave you, you can also add URL parameters in Google Webmaster Tools

https://support.google.com/webmasters/answer/1235687?hl=en

[WesleySmits]

You could add a canonical tag to link to the default page. This way Google will know that it should only index that.
The code for this would be:

This should be placed in the section of your HTML code.

Some more resources on the subject:

• Complete guide to rel canonical
• Learn about canonicalization and rel canonical