Crawl Diagnostics Report

[coremediadesign]

I'm a bit concerned about the results I'm getting from the Crawl Diagnostics Report.

I've updated the site with canonical urls to remove duplicate content and when I check the site - it all displays the right values, but the report, which has just finished crawling is still showing a lot of pages as duplicate content.

Simple example:

http://www.domain.com

http://www.domain.com/

Both of them are in the duplicate content section although both have canonical url set as:

Does each crawl check the entire site from the beginning or just the pages it didn't have a chance to crawl the last time?

This is just one of 333 duplicate content pages, which have canonical url pointing to the right page.

Can someone please explain?

[kenneth_martin]

Yep!

That's why I really like the csv files because you can sort stuff and filter things down to specifically what you want to see.

[coremediadesign]

Hi Kenny,

Thanks for getting back to me.

So is it just the way it is reported on the page and it's not the actual problem with the duplicate content?

[kenneth_martin]

Hi Sebastian,

Sorry for the confusion. Our software currently reports those urls as having both duplicate content and canonical tags.  I find that the best way to view this information is by exporting your crawl diagnostic's csv file. You can easily locate the export functionality in the upper right of the crawl diagnostic page.

Kenny