Duplicate Content - Bulk analysis tool?
-
Hi
I wondered if there's a tool to analyse duplicate content - within your own site or on external sites, but that you can upload the URL's you want to check in bulk?
I used Copyscape a while ago, but don't remember this having a bulk feature?
Thank you!
-
Great thank you!
I'll give both a go!
-
Great thanks
Yes I use screaming frog for this, but it was to look at actual page content. So yes to see if sites copy our content, but also to see whether we need to update our product content as some products are very similar.
I'll check the batch process on copyscape thanks!
-
I have not used this tool in this way, but have used it for other crawler projects related to content clean up and it is rock solid. They have been very responsive to me on questions related to use of the software. http://urlprofiler.com/
Duplicate content search is the project next on my list, here is how they do it.
http://urlprofiler.com/blog/duplicate-content-checker/
You let URL profiler crawl the section of your site that is most likely to be copied (say your blog) and you tell URL profiler what section of your HTML to compare against (i.e. the content section vs the header or footer). URL profiler then uses proxies (you have to buy the proxies) to perform Google searches on sentences from your content. It crawls those results to see if there is a site in the Google SERPs that has sentences from your content word for word (or pretty close).
I have played with Copyscape, but my markets are too niche for it to work for me. The logic here from URL profilers is that you are searching the database that most matters, Google.
Good luck!
-
I believe you might be able to use List Mode in ScreamingFrog to accomplish this, however it depends on ultimately what your goal is to check for duplicate content. Do you simply want to find duplicate titles or duplicate descriptions? Or do you want to find pages with sufficiently similar text as to warrant concern?
== Ooops! ==
It didn't occur to me that you were more interested in duplicate content caused by other sites copying your content rather than duplicate content among your list of URLs.
Copyscape does have a "Batch Process" tool but it is only available to paid subscribers. It does work quite nicely though.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Content - What can be duplicate in two different product pages.
I am having a hard time understanding how my 3 different product pages are being shown up as Duplicate Content in s crawl. Some of my 21 different pages are being shown as duplicate content. Here are 3 of those: 1. http://champu.in/korn-rock-band-mens-round-neck-t-shirt-india 2. http://champu.in/stop-the-burning-mens-round-neck-t-shirt-india 3. http://champu.in/funny-t-shirts/absolut-punjabi-red-men-s-round-neck-t-shirt Can someone help me with this. Thanks in advance 🙂
On-Page Optimization | | sidjain4you0 -
Duplicate content shown in Google webmaster tools for 301 redirected URLs.
Why does Google webmaster tools shows 5 URLs that have been 301 redirected as having duplicate meta descriptions?
On-Page Optimization | | Madlena0 -
Webmaster tools
Hi there, I have access to my sebsite writeing www.piensapiensa.es or www.piensapiensa.com. What domain should I add to the webmasters tools? Should I have to do some kind of 301 direction from piensapiensa.com to piensapiensa.com as the main market is in Spain? Thanks.
On-Page Optimization | | juanmiguelcr0 -
Duplicate content and the Moz bot
Hi Does our little friend at SEOmoz follow the same rules as the search engine bots when he crawls my site? He has sent thousands of errors back to me with duplicate content issues, but I thought I had removed these with nofollow etc. Can you advise please.
On-Page Optimization | | JamieHibbert0 -
Duplicate content on area specific sites
I have created some websites for my company Dor-2-Dor and there is a main website where all of the information across the board is on (www.dor2dor.com) but I also have area specific sites which are for our franchisees who run certain areas around the country (www.swansea.dor2dor.com or www.oxford.dor2dor.com) The problem is that the content that is on a lot of the pages is the same on all of them for instance our faq's page, special offers etc. What is the best way to get these pages to rank well and not have the duplicate content issues and be ranked down by search engines? Any help will be greatly received.
On-Page Optimization | | D2DWeb0 -
Notonthehighstreet.co.uk - duplicate content? a reason to not sell via 3rd parties
A mixture of questions and discussion Question 1. can the following two pages be considered duplicate content http://www.notonthehighstreet.com/gardenbeet/product/deer-head-wall-art http://www.notonthehighstreet.com/1/1/219933-deer-head-wall-art-by-garden-beet.html both pages are indexed and both pages have different meta - aimed at different search combinations Discussion The search for 'deer head wall art gardenbeet' is generated by my PR company - we have done loads of print advertising for this item yet the sheer mass and volume of noths.com stops my store http://www.gardenbeet.com/garden-wall-art/58-deer-head.html from obtaining the number one position. All is fair in the business world I suppose BUT the original marketing machine for noths.com was claiming that they were assisting the small business owner. I paid them over £600 to join and now they compete with me head on. Stupid me I suppose. Let this be a key learning for those toying with the idea of investing in their own SEO or a 3rd party selling platform. Ho hum
On-Page Optimization | | GardenBeet0 -
Website Content
Is it bad to have html pages on a blog? I converted a completely HTML site to wordpress, but havd hundreds of article pages that are still html.
On-Page Optimization | | azguy0 -
Duplicate Page Title Elements
Hello Mozzers. My questions is below and I would like to thank everyone in advance for any feedback 😉 I own a dog supplies site (www.k9electronics.com). When I launched the site several years back I hired a guy for SEO and he optimized my home page for specific categories search terms such as "dog training collars", "dog shock collars:, ect instead of general search terms such as "dog supplies", "dog accessories", ect. I would like to start moving these home page title element terms (starting with "dog shock collars") over to the dog training collars category but have high rankings for this term on the home page. Current Home Page Title Element:
On-Page Optimization | | k9byron
dog training collars, dog shock collars, electric dog collar, dog supplies (recently added) Current Dog Training Collars Category Title Element:
dog training collars I was hoping to add "dog shock collars" to the dog training collars category page until I achieved higher ranking then delete if from the home page. ..or swap it out with "dog accessories". I am currently ranked #5 in Google for "dog shock collars" on the home page & dog training collars category page ...and I am a little concerned about changing these title elements. My question is; If I add 'dog shock collars" to the dog training collars category page title as well, how will it effect my ranking on both pages having this duplicate term in both page titles? Thank You,
Byron-0