Many Duplicate Content Flags
-
Not sure about you all, but I’m loving the new Moz Site Crawler. However, I was noticing that it is identifying a huge amount of pages as duplicate content.
There are about 30,000 pages in this website, with that said we’ve had to make many templates to make the site scalable. Additionally a url rule was lost which caused a significant amount of duplicate pages to be created. I am working through using the moz crawl tool to identify duplicate pages but noticing many pages under “Affected Pages,” are actually unique content pages with initial content that is duplicate.
I read that Moz flags any pages with 90% or more content overlapping content or code. My theory for this is that some templates that are too similar, to the point that Moz reads them as duplicative. Has this happened for anyone else?
In addition, if Moz is flagging these similar pages as duplicate content, do we surmise that Google bots are having the same issue? We have seen issues with rankings as it pertains to the actual duplicate pages but hadn't experienced issues across the unique pages, they are hyperlocal pages so we are able to see rankings quite easily.
-
Hey there,
Sam from Moz's Help Team here!
You're correct - our tool has a 90% tolerance for duplicate content, which means it will flag any content that has 90% of the same code between pages. This includes all the source code on the page and not just the viewable text.
You can run your own checks for percentage similarity using this tool: http://smallseotools.com/similar-page-checker/.
If we're identifying two pages as having duplicate content it's likely that Google will be running into the same issues. You can read a little more about duplicate content, and how to resolve it, on our resource page here: https://moz.com/learn/seo/duplicate-content
Let us know if we can help with anything else!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is MOZ any good to analyze an e-commerce site? How come that a cms page can be seen as duplicate content with a category page?
Hi Guys, I've been using Moz for quite a long time now for 2 of my shops. Now I am in the process of launching the second shop and I just don't understand how is it possible that a cms static page (About US) to be seen as a duplicate content with other 96 pages - including product pages and other totally different pages such as delivery information, category pages, returns and so on. Really MOZ?? Is it me or you?? Your help would be much appreciated! Thank you!
Moz Bar | | Sorin_T0 -
I keep getting a 429 Too Many Request error for a wp-login page on my website. Is there a way to prevent that from happening, or fix outside of redirecting on the back end of WordPress?
I have a client that keeps coming up with 429 Too Many Request critical crawl errors. I re-directed some of them on the back end of WordPress, but additional keep coming in. The URL has a WP-Login and directs to back end login section. Is there a reason that would come up as an error, how can I prevent it from happening again, and how can I fix the remaining current errors outside of redirecting back to /? Thanks, Kalyn Lengieza
Moz Bar | | GrindstoneConsult0 -
Are there any tricks for checking duplicate content?
Hello all, My MOZ weekly scan keeps coming back with indication for duplicate content on pages that don't have that much alike. I feel like I must be missing something. Is there any place I can plug in two urls so that it would tell me what the similarities are and figure out how to make it less of a duplicate content? https://www.stage32.com/happy-writers/pitch-sessions/Pitch-Amar-Hansen-Saturday-October-29th-2016
Moz Bar | | stage32
and
https://www.stage32.com/happy-writers/pitch-sessions/Pitch-Will-Raynor-Wednesday-November-16th-2016 OR https://www.stage32.com/webinars/Film-Contracts-101-Everything-You-Need-To-Know
and
https://www.stage32.com/webinars/Breaking-Down-IP-Intellectual-Property-For-Development Any ideas? Thanks in advance.0 -
Moz Content --- for SEO or simply user engagement?
What is the primary function of Moz Content? It looks like it is most useful for managing content as a user engagement tool. Our content strategy is centered on boosting organic placement - with user engagement as a nice but unessential side product. Besides providing general descriptive details of a site's content / authorship - how can Moz Content help with SEO?
Moz Bar | | cvonhassell0 -
Site Crawl report show strange duplicate pages
Beginning in early in Feb, we got a big bump in duplicate pages. The URLs of the pages are very odd: Example URL:
Moz Bar | | Neo4j
http://firstname.lastname@website.com/dir/page.php
is duplicate with http://website.com/dir/page.php I checked though the site, nginx conf files, and referral pages, and could not find what is prefixing the pages with 'http://firstname.lastname@'. Any ideas? The person whose name is 'Firstname Lastname' is stumped as well. Thanks.0 -
Duplicate Page Titles detected, no relevant links shown
Moz is reporting duplicate page titles, but the relevant pages aren't being shown. The downloaded report doesn't show them either. Screencast: http://screencast.com/t/gkyRds6u
Moz Bar | | ElykInnovation0 -
Duplicate page content
The MOZ crawler identifies pages as duplicate content which are not the same.
Moz Bar | | aignerart
The pages http://www.aignerart.com/abstracts-oil-painting/cicli-colora.html and http://www.aignerart.com/abstracts-oil-painting/murs-de-la-ville.html are marked duplicate but they are different paintings. Any ideas?0 -
Moz Crawler URL paramaters & duplicate content
Hi all, this is my first post on Moz Q&A 🙂 Questions: Does the Moz Crawler take into account rel="canonical" for search results pages with sorting / filtering URL parameters? How much time does it take for an issue to disappear from the issues list after it's been corrected? Does it come op in the next weekly report? I'm asking because the crawler is reporting 50k+ pages crawled, when in reality, this number should be closer to 1000. All pages with query parameters have the correct canonical tag pointing to the root URL, so I'm wondering whether I need to noindex the other pages for the crawler to report correct data?: Original (canonical URL): DOMAIN.COM/charters/search/mx/BS?search_location=cabo-san-lucas Filter active URL: DOMAIN.COM/charters/search/mx/BS?search_location=cabo-san-lucas&booking_date=&booking_days=1&booking_persons=1&priceFilter%5B%5D=0%2C500&includedPriceFilter%5B%5D=drinks-soft Also, if noindex is the only solution, will it impact the ranking of the pages involved? Note: Google and Bing are semi-successful in reporting index page count, each reporting around 2.5k result pages when using the site:DOMAIN.com query. The rel canonical tag was missing for a short period of time about 4 weeks ago, but since fixing the issue these pages still haven't been deindexed. Appreciate any suggestions regarding Moz Crawler & Google / Bing index count!
Moz Bar | | Vukan_Simic0