Duplicate Content Report: Duplicate URLs being crawled with "++" at the end
-
Hi,
In our Moz report over the past few weeks I've noticed some duplicate URLs appearing like the following:
Original (valid) URL:
http://www.paperstone.co.uk/cat_553-616_Office-Pins-Clips-and-Bands.aspx?filter_colour=Green
Duplicate URL:
http://www.paperstone.co.uk/cat_553-616_Office-Pins-Clips-and-Bands.aspx?filter_colour=Green**++**
These aren't appearing in Webmaster Tools, or in a Screaming Frog crawl of our site so I'm wondering if this is a bug with the Moz crawler? I realise that it could be resolved using a canonical reference, or performing a 301 from the duplicate to the canonical URL but I'd like to find out what's causing it and whether anyone else was experiencing the same problem.
Thanks,
George
-
So glad to help, George!
-
Hi Chiaryn,
Thanks - you've been really helpful! I had assumed that as the referrer wasn't in the Web UI (per WMT), it wasn't available anywhere. I'd also assumed it was a copywriting issue and not a product data issue.
Need to readdress my assumptions
George
-
Hey George,
Thanks for writing in.
I looked into the pages with the ++ in the URL and it seems that they do actually exist on the site, so it isn't an issue with our crawler that is causing these in your crawl errors. For example, a link to the URL http://www.paperstone.co.uk/cat_553_Desktop-Essentials.aspx?filter_colour=Green++ can be found in the source code of the page http://www.paperstone.co.uk/cat_553_Desktop-Essentials.aspx here: http://screencast.com/t/HpHTlSs5gH8H
You can find the referral pages for the ++ pages on the site by downloading the Full Crawl Diagnostics CSV. In the first column, perform a search for the ++. When you find the correct row, look in the column labeled referrer, AM. This tells you the referral URL of the page where our crawlers first found the URLs that include ++. You can then visit this URL to find the links to those pages.
Since these URLs with the ++ do resolve with a 200 http status and they have the same code and content as the pages without the ++, our crawler will count them as duplicate content. I'm not certain why Screaming Frog and GWT are not find or reporting these pages; it may be that they parse the + signs in the URL differently than our crawler does.
As Keri and bishop23 mentioned, this is most likely not a major issue if GWT isn't reporting the errors, but we prefer to report the issues because we would rather be safe than sorry.
I hope this helps. Please let me know if you have any other questions.
Chiaryn
-
I'm not seeing an answer that jumps out at me for this one. For the immediate future, don't sweat it if you're not seeing it in GWT. This is assigned to our help desk, and we'll have someone from there investigate more and get back to you, though it might be a few days because of the Thanksgiving holiday (if you don't get an answer today, it may be Monday before we have a chance to respond).
-
If they're not appearing on WMT than you should ignore unless it's an exact duplicated content, then delete
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Solved Page content writing tool
A long time ago I'm pretty sure I used a tool that helped me write content for a webpage. It checked my writing as I typed and gave suggestions. I can't find the tool anymore. I hope I didn't imagine it as it was really useful..
Product Support | | laurentjb0 -
I just wanted to have custom URL for my MOZ usr profile and was unable to get done. Is there any one aware about this?
Looking for an answer regarding the community awareness about this implementation.
Product Support | | Sanka_Liyanage0 -
Is there a report that will show Keyword Rank History for the life of an active campaign?
I want to get a history of the keywords that have been tracked in an active campaign. This will allow us to see improvements over time.
Product Support | | liftedlogic1 -
Duplicated content issue?
Hi, Moz tools tell me that I have approx. 15xx Pages with High Priority Issues [Duplicate Page Content] going thru the details here are the majority: many products no longer exist and visitor redirected (404) to a same not found page. duplicated content with url appended with utm_source/utm_medium/utm_campaign GA tracking paras. On those page there are already canonical tags plus I've configed those parameters in Google Webmaster "URL Parameters" how much of these warning could compromise the seo proformance or provided that there have been dealt with should I just ignore these warning?
Product Support | | LauraHT0 -
I'd like to "pause" my account for a few months without canceling it. Is there a way to do so?
I'd like to "pause" my account for a few months without canceling it. Is there a way to do so?
Product Support | | Akufnx0 -
How to forward a report, email
I am trying to send the full report on my site, www.torontopainters.org for them to examine it but apparently I forwarded the email but it was not useful it was just the first page or something. Thanks Vic
Product Support | | oakvillePainters0 -
How to get in contact with moz about my report content?
Just logged into moz to view my reports as i got an email about a new report went into my dashboard and nothing is coming up. Is there any way i can get in direct contact with moz?
Product Support | | meteorelectrical0