Checking for content duplication against content on your own site.
-
We are currently trying to rewrite our product descriptions and I'm afraid some of the salespeople that are writing the descriptions are plagiarizing one-another's writing. Is there a content duplication checker that will allow you to check a piece of writing against a specific site rather than all of the web?
-
I assume that you have an admin section in the CMS where you are editing and entering these articles before they go live.
You need to get a developer to simply write a search algo that when you create a new article and before it goes live, it takes sections of your content and looks for matches/duplicates. You can set a requirement that it has to match on a minimum of a 4 to 5 word string and other such limitations to make sure you are not matching too many items. It will take a few tests to find a sweet spot of too many matches vs not enough.
With 17K pages, this is the only way you can really do this in an efficient way, you need some IT support/development. They may have to create a reporting layer as well to help you sift through the results.
Good luck.
-
I have two dev servers, one of which it is possible to do what you're talking about but that is the absolute least efficient tool to use for this.
The crawl diagnostics are updated about once a week which means I would have to post the new content and hope I got it online in time for the crawl. If I didn't then I would have to wait an additional week to see results.
The crawl diagnostics also limits the amount of pages it will crawl on your site to 10,000. I stated before that I have over 17,000 pages. So even if I did use this method, the chances of that page being crawled is little better than 50/50.
Also, the crawl diagnostics only tell you what pages have duplicate content - not the exact content that was duplicated. That means I'd have to manually find the page I'm targeting, then follow the supposed duplicate content suggestions proposed by the crawler and find the similarities myself.
I think it's very safe to say that the crawl diagnostics, nor any product that SEOmoz provides, is an answer to my issue. If I thought it was, I would have already been using it and would not have posted this question.
-
Hi Michael,
Having a website that big means that you might have a test or dev environment.
If not create one.
if you have something like test.yourwebsite.com and submit it to the SEOmoz tools as a new project you can see a report before your website goes live.
Cornel
-
Those are good answers and would work on a smaller scale site. We currently have over 17,000 product pages so I can't really use either method. It's looking like a google custom search is the best bet even though I can't search an entire paragraph at a time.
-
Just off the top of my head, there are a few low tech ways to do it....
If you have Win 7 the searching has improved greatly - just move all files to a local machine - and search the directory you placed in for the content you are wanting to check - it will give all files that contain the words. (but can become overloading)
If you have dreamweaver or other enterprise level editor - almost all have a site search function to where you can search/profile code/text and have it find one by one which pages contain the searched terms - or globally list them.
Other than that, probably a custom script -or a google search for an HTML profiler might help?
Shane
-
That's for pages that are already published and crawled. I want to able to search my site for entire sentences and/or paragraphs of text that I have yet to publish so I can make sure it's not being used elsewhere on the site. The crawl diagnostics tell me I have duplicate content after the fact - I'm trying to take a proactive approach rather than reactive.
-
The duplicate content from you website is shown in the SEOmoz tools.
Check the Crawl Diagnostics Summary:
Cornel
-
That site searches the entire web for copies. I'm looking for something to crawl my own site for duplicate content.
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ecommerce product page duplicate content
Hi, I know this topic has been covered in the past but I haven't been able to find the answers to this specific thing. So let's say on a website, all the product pages contain partial duplicate content - i.e. this could be delivery options or returning policy etc. Would this be classed as duplicate content? Or is this something that you would not get concerned about if it's let's say 5-10% of the content on the page? Or if you think this is something you'd take into consideration, how would you fix it? Thank you!
On-Page Optimization | | MH-UK0 -
Duplicate Content
I run a Business Directory, Where all the businesses are listed. I am having an issue.. with Duplication content. I have categories, Like A, B, C Now a business in Category A, User can filter it by different locations, by State, City, Area So If they filter it by the State and the State has 10 businesses and all of them are in one City. Both of the page
On-Page Optimization | | Adnan4SEO
The state filtered and the city filtered are same, What can i do to avoid that? canonical-url-tag or changing page Meta's and Body text? Please help 🙂0 -
Duplicate Content Issue in Magento
Hi I need help in resolving the duplicate content issue on my magento site I got a product My main product url is https://www.oakfurnitureking.co.uk/shop-by-product/boston-solid-oak-4-drawer-chest and it got variation of url see below that are causing duplicate content issue , I have inserted the canonical tag on the below url and my main url is https://www.oakfurnitureking.co.uk/shop-by-product/boston-solid-oak-4-drawer-chest but still moz is showing it as duplicate content. Help Please <colgroup><col width="1003"></colgroup>
On-Page Optimization | | Adnan.Hassan.Khan
| https://www.oakfurnitureking.co.uk/product/oak-bedroom-furniture/boston-solid-oak-4-drawer-chest |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/6/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/17/ |
| https://www.oakfurnitureking.co.uk/shop-by-range/boston/boston-solid-oak-4-drawer-chest |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/42/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/63/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/67/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/46/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/79/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/88/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/75/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/90/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/92/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/33/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/27/ |
| https://www.oakfurnitureking.co.uk/shop-by-range/boston-solid-oak-4-drawer-chest |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/50/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/22/ |
| https://www.oakfurnitureking.co.uk/catalog/product/view/id/45/s/boston-solid-oak-4-drawer-chest/category/74/ |0 -
Duplicate Content only an Issue on a Huge Scale?
To what extent is duplicate content an issue? We have a support forum with some duplicate content because users ask the same questions. The Moz reports we receive highlights our duplicate content and page title for our support forum as a "big" issue. I'm unsure to what extent it harms our SEO, and making the support section non-crawable would impair our level of support. It would be nice to know for sure if we should be concerned about this, and if yes, how can we do it differently? Thanks, I appreciate you help. -Allan
On-Page Optimization | | Todoist0 -
Duplicate Content from WordPress Category Base?
I recently changed my category base in WordPress and instead of redirecting or deleting the old base, WordPress kept the content up. So I now have duplicate content on two different urls - one on the old category base, one on the new category base. How should I handle this situation? The site is only a couple weeks old, if that makes any difference.
On-Page Optimization | | JABacchetta0 -
Checking Duplicate Content
Hi there, We are migrating to a new website, which we are writing lots of new content for the new website. The new website is hosted on a development site which is password protected and so on so that it cannot be indexed. What i would like to know is, how do i check for duplicate content issues out there on the world wide web with the dev site being password protected? Hope this makes sense. Kind Regards,
On-Page Optimization | | Paul780 -
Duplicate content on homepage?
Hi I have just created a new campaign and it states that I have duplicate page content which would affect search rankings. Basically it is counting my site www.mydomain.com and www.mydomain.com/index.php as two seperate pages. How can I make it so that only www.mydomain.com is visible reducing the duplicate content issue? Many Thanks
On-Page Optimization | | idv0 -
Duplicate content issue with dynamically generated url
Hi, For those who have followed my previous question, I have a similar one regarding dynamically generated urls. From this page http://www.selectcaribbean.com/listing.html the user can make a selection according to various criteria. 6 results are presented and then the user can go to the next page. I know I should probably rewrite url's such as these: http://www.selectcaribbean.com/listing.html?pageNo=1&selType=&selCity=&selPrice=&selBeds=&selTrad=&selMod=&selOcean= but since all the results presented are basically generated on the fly for the convenience of the user, I am afraid google my consider this as an attempt to generate more pages as there are pages for each individual listing. What is my solution for this? Nofollow these pages? Block them thru robots txt?
On-Page Optimization | | multilang0