Welcome to the Q&A Forum

Hi Tom

Short answer is that it probably won't be a problem. From what you are saying the duplication is 'natural' in the sense it is information which you might normally expect to see duplicated since it relates to similar products across multiple sites (also think privacy policy or terms and conditions pages).  In this case it is unlikely to attract a penalty.

Matt Cutts covered that topic in this video (posted on SearchEngineLand): Duplicate Content Won't Hurt You Unless its Spammy.

However, it will probably mean you are leaving it up to the search engine to decide which 'version' of your duplicate content it should prioritise and serve up to people searching. If it is not important to rank for the content on these duplicate pages then again it is not really an issue.

However if you want to play safe or aim to get some rankings for a specific page - among all the duplicate versions, you can use a rel=canonical tag to let the search engines know which page is the "original" so that they will prioritise this one (ie point the link juice at a specific page). Matt Cutts talks about that in this video (although he talks about it in the context of a news article).

Check out this Moz article on Duplicate Content - it also has a short explanation on how to use the rel=canonical.

All the best

Neil

Hi Laura

I am not sure that you can use robots.txt to prevent a search engine bot from crawling a part of a page. Robots.txt is usually used to exclude a whole page.

The effect of the duplicate content on your search engine optimisation depends in part on how extensive is the duplication. In many cases it seems that Google won't penalise the duplicate content (it understands that some content will of necessity be duplicated) - see this video by Matt Cutts from Google.

Duplicate Content is Small (Short Paragraph)

From your question it sounds like you are talking about part of page and it sounds like a relatively small part - I assume you are not a shipping company so the shipping info would be a small part of the page.

In which case it may not affect your search engine optimisation at all (assuming you are not trying to rank for the shipping info).

As long as the content on the rest of the page is unique or different from other pages on the site.

Duplicate Content is Large (but not a page)

If the shipping info is substantial (say a couple of paragraphs or half the content on the page) then Google suggests you create a separate page with the substantial info on it and use a brief summary on other pages with a link to the separate page:

• Minimize boilerplate repetition: For instance, instead of including lengthy copyright text on the bottom of every page, include a very brief summary and then link to a page with more details. In addition, you can use the Parameter Handling tool to specify how you would like Google to treat URL parameters.

(from Google Webmaster: Duplicate Content)

Duplicated Pages

Much of the discussion about duplicated content is more about whole pages of duplicated content. The risk with these pages are that search engines may not know which to rank (or more to the point rank the one you don't want to rank). This is where you might use a rel=canonical tag or a 301 redirect to direct or hint to the search engine which page to use.

Moz has a good article on Duplicate Content.

All the best

Neil

Hi  T and RafalJ

Thanks for your question and reply - it got me doing some more research and I followed RafalJ's lead (link). It was not clear to me how RafalJ's solution solved the initial problem which was avoiding duplicate content problems but now I see how it works.

I found this question has come up before on Moz: Geotargeting Duplicate Content to Different Regions (http://moz.com/community/q/geotargeting-duplicate-content-to-different-regions-href-and-canonical-tag-confusion) and includes a link to this article in Dejean SEO (http://dejanseo.com.au/canonical-vs-hreflang/) (recommended by the ubiquitous Gianluca Fiorelli).

The DejeanSEO articles provides another angle in support of RafalJ's suggestion.

cheers

Neil

(sorry about the naked links - typing on my ipad and the links popup does not play nice)