Moz Crawler not Identifying all Duplicate Pages
-
On two recent site crawls (9/27/14 and 11/4/14) for duplicate content the Moz tool did not ID the following 2 pages, which are 100% duplicate to each other:
http://www.hooksandlattice.com/planter-hampton-241212.html ; Screenshot: http://screencast.com/t/DdwWroUU
http://www.hooksandlattice.com/planter-hampton-721212.html ; Screenshot: http://screencast.com/t/8Lb1cJZmGrhX
As I'm working feverishly to re-write and update the site (goal is ZERO duplicates) I'm finding it challenging to use the Moz tool to get the project done. Does anyone have any feedback or help they can provide for how I can identify all duplicate pages associated with my domain?
Thank you!
Lindsey Pfeiffer
-
Hi Lindsey
Our engineers have confirmed that rogerbot will flag pages that are 100% identical but can sometimes miss pages that are 99% similar. The crawler is deliberately written to err on the side of not reporting false positives which means it sometimes can report false negatives which has occurred in your case. Using a combination of tools such as Webmaster tools can help isolate any pages we have missed.
Hope this helps!
-
Hey Lindsey!
I am not sure why our crawler did not flag those pages as they are 99% identical and are not sharing the same canonical URL. This is very strange and I'll send this up to our crawler engineer to obtain more insight.
Will let you know what I find out once I hear back!
-
Do you check Google Webmaster Tools? Under Search Appearance > HTML Improvements Google will list duplicate titles and descriptions among other things, which might be a help to you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Looking for a Tool to Find Referring Pages of Specific URLs
Hello Everyone, We are Looking for a Tool to Find Referring Pages of Specific URLs. Please let me know if you know of a Moz tool or another tool for this need. Thanks.
Moz Bar | | Pushm4 -
Title tag not registering on page optimization
I added a link to the "page optimization" section at moz. When I submit the url to be reviewed (click "optimize") it doesn't find the title tag, although it is definitely there e.g this page
Moz Bar | | digitalalchemy
http://naamanewman.co.il/photosGalleryID.aspx?id=34 Anybody as any Idea why is that? Maybe the tag placement on the DOM (it's not directly under the opening head tag).0 -
Comparing Competitor On-Page Grades?
When I run an advanced SERP report or even the basic keyword difficultly report it gives me competitor On-page grades. I can look directly at their code, but I'm wondering if there is a way to easily see and export a .csv or excel file of the on-page ranking factors for the top 1-10 results comparing them to a page on my site? Seems like that would be really helpful and much quicker than looking at source code to understand what on-page factors are contributing to their success. Is there any way to do this in Moz or a free tool elsewhere you would suggest?
Moz Bar | | K-WINTER0 -
Using Moz to find blogger with good PR.
Hi, My manager has recently informed me that you can use Moz in order to find blogger by category and their page rank. Is this possible? The reason for this is because we are looking for blogger with a good page rank and who are relevant to our sector, which is days out, outdoor activities, zipwires, adventure etc. We then want to contact them to see if we can invite them to write about our Tree Top Adventure and zip trekking activities. Any help would be much appreciated.
Moz Bar | | GoApe_20140 -
On Page Grader can't access my URLs
HI- I am trying to grade some specific pages for keywords with the on page grader but it keeps telling me "Sorry, but that URL is inaccessible. " I can reach them via the browser and they are not https. Any thoughts? Here is a sample: www.bulkcandystore.com/kosher-candy Any help is appreciated. Ken
Moz Bar | | CandymanKen0 -
Clarify "broad keyword usage in page title"
Hello Page grader has two different grades for page title that I want clarification on. There is "Broad Keyword Usage in Page Title" and "Exact Keyword Usage in Page Title". Googling around about and searching here I have found that "broad" seems to mean the keywords should be used throughout the page, rather than just in the title and header. Which makes sense as this is a kind of check to ensure the page IS about the keywords and not something unrelated. But what is meant by "broad" usage in the page title? This refers specifically to the page title and not the whole document. My best guess got me to this, given the keyword "Visit London Today"; "Come and visit London today" - exact match only "London - visit today" - broad match only "Visit London the city of dreams | visit London today" - matches both That could be complete nonsense, but basically is broad usage the use of keywords scattered in the page title? Thanks.
Moz Bar | | yolkcreative0 -
Does Moz Pro generate similar keyword phrases in a list (preferably showing their difficulty %) or is it only one phrase at a time with no similar words/phrases suggested?
I just signed up for Moz Pro but the keyword research seems to only let you try one keyword phrase at a time. Is there a way for it to give related keywords along with their difficulty % info, etc. It is far too slow and inconvenient doing one at a time.
Moz Bar | | SavingSpotlight0 -
Can I delete a SEO campaign in Moz, and start a new one for a different website?
Would be nice to know, as I'm limited to 5 campaigns, and the most important work is done for 2 site's, so I would like to switch that to 2 other website's. Regards,
Moz Bar | | mrblue910