Many Duplicate Content Flags
-
Not sure about you all, but I’m loving the new Moz Site Crawler. However, I was noticing that it is identifying a huge amount of pages as duplicate content.
There are about 30,000 pages in this website, with that said we’ve had to make many templates to make the site scalable. Additionally a url rule was lost which caused a significant amount of duplicate pages to be created. I am working through using the moz crawl tool to identify duplicate pages but noticing many pages under “Affected Pages,” are actually unique content pages with initial content that is duplicate.
I read that Moz flags any pages with 90% or more content overlapping content or code. My theory for this is that some templates that are too similar, to the point that Moz reads them as duplicative. Has this happened for anyone else?
In addition, if Moz is flagging these similar pages as duplicate content, do we surmise that Google bots are having the same issue? We have seen issues with rankings as it pertains to the actual duplicate pages but hadn't experienced issues across the unique pages, they are hyperlocal pages so we are able to see rankings quite easily.
-
Hey there,
Sam from Moz's Help Team here!
You're correct - our tool has a 90% tolerance for duplicate content, which means it will flag any content that has 90% of the same code between pages. This includes all the source code on the page and not just the viewable text.
You can run your own checks for percentage similarity using this tool: http://smallseotools.com/similar-page-checker/.
If we're identifying two pages as having duplicate content it's likely that Google will be running into the same issues. You can read a little more about duplicate content, and how to resolve it, on our resource page here: https://moz.com/learn/seo/duplicate-content
Let us know if we can help with anything else!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Content & Title Tag Group Fields on MoZ Report
Hello, On my SEO MOZ exported Site Crawl CSV report, I have columns for Duplicate Content Group & for Duplicate Title Tag Group. The values in the columns are numerical - 20, 5 , 15, etc. Can anyone explain to me what these values represent and how I can fix the issues I presume they represent? Thank you,
Moz Bar | | AED-1
Scott0 -
How Can I Batch Upload URLs to get PA for many pages?
Howdy folks, I'm using advanced search operators to generate lists of long tail queries related to my niche and I'd like to take the batch of URLs I've gathered and upload them in a batch so I can see what the PA is for each URL. This would help me determine which long tail query is receiving the most love and links and help inform my content strategy moving forward. But I can't seem to find a way to do this. I went to check out the Moz API but it's a little confusing. It says there's a free version, but then it looks like it's actually not free, then I try to use it and it says I've gone over my limit even though I haven't used it yet. Anyone that can help me with this, I'd really appreciate it. If you're familiar with SEMRush, they have a batch analysis tool that works well, but I ideally want to upload these URLs to Moz because it's better for this kind of research. Thanks!
Moz Bar | | brettmandoes2 -
Duplicate content found in scan
On June 8th we ran a Moz Crawl on our site. We found 144 pages that were flagged with duplicate content.
Moz Bar | | StickyLife
Again on June 13th we ran another moz crawl on our site and found 137 pages that were flagged with duplicate content. Then one final scan on June 22nd with 161 pages of duplicate content. After comparing the 3 different scans I see that, without making any changes, pages that were not flagged as duplicate content are now being flagged as duplicate content. While at the same time, pages that were originally flagged as duplicate content are now no longer showing up with duplicate content. I could understand if we made some changes to these pages but no changes were made. For example: On the 8th this page was flagged as duplicate content - https://www.stickylife.com/star-magnet
On the 13th and 22nd it was not flagged as duplicate content but no changes were made to that page. For reference it was flagged as duplicate content with the following page: https://www.stickylife.com/baseball-glove-magnet This page was also Not changed or altered between between these dates. In addition, when Moz scans our site through our campaign every Friday the results do not match what we see when we do a manual scan. Moz's weekly scan only reveals 14 pages with duplicate content as opposed to the numbers you see above. Why such inconsistencies in the Moz Scans?0 -
How can I find duplicate pages from a Moz Crawl?
We have many duplicate pages that show up on the Moz Crawl, and we're trying to fix these but it's very difficult because I can't see a way to isolate the code where the duplicate is found. For instance, http://experiencemission.org/immersion/ is one of our main pages, and the crawl shows one duplicate of http://experiencemission.org/immersion. It appears that one of our staff manually edited the source code in one of our pages but forgot the trailing slash. This would be an easy fix but the problem is that this page is linked to internally on our website 2423 times, so it's next to impossible to find the code that is incorrect. We have many other pages with this same basic problem. We know we have duplicates, but it's next to impossible to isolate them. So my question is this: When viewing the Moz Crawl data is there any way to see where a specific duplicate page link is located on our website? Thanks for any and all help!
Moz Bar | | expmission0 -
Moz Crawler not Identifying all Duplicate Pages
On two recent site crawls (9/27/14 and 11/4/14) for duplicate content the Moz tool did not ID the following 2 pages, which are 100% duplicate to each other: http://www.hooksandlattice.com/planter-hampton-241212.html ; Screenshot: http://screencast.com/t/DdwWroUU http://www.hooksandlattice.com/planter-hampton-721212.html ; Screenshot: http://screencast.com/t/8Lb1cJZmGrhX As I'm working feverishly to re-write and update the site (goal is ZERO duplicates) I'm finding it challenging to use the Moz tool to get the project done. Does anyone have any feedback or help they can provide for how I can identify all duplicate pages associated with my domain? Thank you! Lindsey Pfeiffer
Moz Bar | | CMC-SD0 -
Weekly Custom Reports Send Duplicates
For one of my sites, I have set up a weekly custom report to be sent out, but when the report comes in, there are multiple copies of the report. Any help would be appreciated on how to make sure that only one copy of the report is sent.
Moz Bar | | Wharthog0 -
Duplicate page issue
Hi Guys We were recently having trouble with an excessive amount of duplicate page titles. So I asked our web company, at a reasonable expense, to fix the issue which they did. How ever I since note that the issue has returned (please see the attached graph. Could anyone explain to me why this might have happened? I t would be great to have some insight before i go back to them. Thanks again for your help Regards Pete Capture.jpg
Moz Bar | | Hardley1110 -
Duplicate Page Content Report on MOZ
Hi, I am just wondering as to the accuracy of this report - does it pick up all the duplicate on page content? Or is there a limit? We have an ecommerce store with a lot of copied and pasted descriptions - just wondering if there is a limit on how much the moz crawler picks up? In other words, once we fix what MOZ has detected, will there be more detected because it is limited to display say up to 200?? Hope you understand what I mean. Thanks
Moz Bar | | bjs20100