Moz Crawler not Identifying all Duplicate Pages
-
On two recent site crawls (9/27/14 and 11/4/14) for duplicate content the Moz tool did not ID the following 2 pages, which are 100% duplicate to each other:
http://www.hooksandlattice.com/planter-hampton-241212.html ; Screenshot: http://screencast.com/t/DdwWroUU
http://www.hooksandlattice.com/planter-hampton-721212.html ; Screenshot: http://screencast.com/t/8Lb1cJZmGrhX
As I'm working feverishly to re-write and update the site (goal is ZERO duplicates) I'm finding it challenging to use the Moz tool to get the project done. Does anyone have any feedback or help they can provide for how I can identify all duplicate pages associated with my domain?
Thank you!
Lindsey Pfeiffer
-
Hi Lindsey
Our engineers have confirmed that rogerbot will flag pages that are 100% identical but can sometimes miss pages that are 99% similar. The crawler is deliberately written to err on the side of not reporting false positives which means it sometimes can report false negatives which has occurred in your case. Using a combination of tools such as Webmaster tools can help isolate any pages we have missed.
Hope this helps!
-
Hey Lindsey!
I am not sure why our crawler did not flag those pages as they are 99% identical and are not sharing the same canonical URL. This is very strange and I'll send this up to our crawler engineer to obtain more insight.
Will let you know what I find out once I hear back!
-
Do you check Google Webmaster Tools? Under Search Appearance > HTML Improvements Google will list duplicate titles and descriptions among other things, which might be a help to you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I keep getting a 429 Too Many Request error for a wp-login page on my website. Is there a way to prevent that from happening, or fix outside of redirecting on the back end of WordPress?
I have a client that keeps coming up with 429 Too Many Request critical crawl errors. I re-directed some of them on the back end of WordPress, but additional keep coming in. The URL has a WP-Login and directs to back end login section. Is there a reason that would come up as an error, how can I prevent it from happening again, and how can I fix the remaining current errors outside of redirecting back to /? Thanks, Kalyn Lengieza
Moz Bar | | GrindstoneConsult0 -
That URL is inaccessible Moz grader?
Hi all, I'm having some issues getting my site graded www.balihaiphoto.com + Kauai wedding photographer with On Page grader, where as when I enter another photographer example www.jmoellerphoto.com and getting results. Is there any reason this is going on that I can correct? Many thanks for any help all! -Jon
Moz Bar | | Jon_Gibb0 -
Re On-Page Grader
One of the pages I'm trying to optimise is achieving an 'A' grade, however all the ticks are black not green as I've seen on other page grade. Why is this? Help much appreciated. Thanks
Moz Bar | | seoman100 -
Moz Tool Bar Annoyance - How do I make the green keyword difficulty box go away?
A highly annoying light green keyword difficulty box just started showing up to the right of the search box. How do I make this go away? ulBzP9W
Moz Bar | | JenKeller0 -
Suggestion: Moz Domain Authority should take disavow into account
Since Moz is trying to predict how Google ranks your site, and Google claims to take the disavow file into account, I'd like to suggest that Moz allow webmasters to upload their disavow file. I imagine this data would be useful to Moz in determining Domain Authority (they may even think of other ways to use it and might even help come to a conclusion on the great debate) and it gives a chance for sites to improve their Moz DA when they are bombarded by spammy links. I'd love to hear the community's thoughts on this idea, as well as the what Wizards of Moz have to say.
Moz Bar | | YairSpolter1 -
MOZ crawl test is not reporting on all the pages on my site.
I've run the crawl test one of the sites I've taken over SEO for, however its only picking all the pages. For instance it indexes all the pages under xxxxx/us but none under xxxxx/au or xxxxx/uk The pages are being indexed as they're ranking in Google. Thanks.
Moz Bar | | ahyde0 -
On Page Grader Question
Hi there, I have just used the On Page Grader on this page: -removed- and checked the grade for the following keywords together: wedding venues bath The On Page Grader has found 15 instances on the page for this keyword. When i view the page source and ctrl-f and search for ''wedding venues bath'', it only finds 8. Is there an issue with the On Page Grader atm?
Moz Bar | | jennie.evans0 -
Connect to Google Analytics problem (Moz Analytics)
Hello - I recently received the Beta invite to Moz Analytics (very excited). I set up the campaign for our primary site and entered our Google Analytics profile - however, Moz does not seem to accept the profile when I click the "Save" button. All of our profiles are listed w/in the drop-down menu, but after selecting the profile we wish to link/connect to - nothing happens. So we're not receiving any stats w/in Moz from our GA acct. Please help. I'm really anxious to begin delving into Moz Analytics! Thanks
Moz Bar | | ryankish0