Duplicate Content Report: Duplicate URLs being crawled with "++" at the end
-
Hi,
In our Moz report over the past few weeks I've noticed some duplicate URLs appearing like the following:
Original (valid) URL:
http://www.paperstone.co.uk/cat_553-616_Office-Pins-Clips-and-Bands.aspx?filter_colour=Green
Duplicate URL:
http://www.paperstone.co.uk/cat_553-616_Office-Pins-Clips-and-Bands.aspx?filter_colour=Green**++**
These aren't appearing in Webmaster Tools, or in a Screaming Frog crawl of our site so I'm wondering if this is a bug with the Moz crawler? I realise that it could be resolved using a canonical reference, or performing a 301 from the duplicate to the canonical URL but I'd like to find out what's causing it and whether anyone else was experiencing the same problem.
Thanks,
George
-
So glad to help, George!
-
Hi Chiaryn,
Thanks - you've been really helpful! I had assumed that as the referrer wasn't in the Web UI (per WMT), it wasn't available anywhere. I'd also assumed it was a copywriting issue and not a product data issue.
Need to readdress my assumptions
George
-
Hey George,
Thanks for writing in.
I looked into the pages with the ++ in the URL and it seems that they do actually exist on the site, so it isn't an issue with our crawler that is causing these in your crawl errors. For example, a link to the URL http://www.paperstone.co.uk/cat_553_Desktop-Essentials.aspx?filter_colour=Green++ can be found in the source code of the page http://www.paperstone.co.uk/cat_553_Desktop-Essentials.aspx here: http://screencast.com/t/HpHTlSs5gH8H
You can find the referral pages for the ++ pages on the site by downloading the Full Crawl Diagnostics CSV. In the first column, perform a search for the ++. When you find the correct row, look in the column labeled referrer, AM. This tells you the referral URL of the page where our crawlers first found the URLs that include ++. You can then visit this URL to find the links to those pages.
Since these URLs with the ++ do resolve with a 200 http status and they have the same code and content as the pages without the ++, our crawler will count them as duplicate content. I'm not certain why Screaming Frog and GWT are not find or reporting these pages; it may be that they parse the + signs in the URL differently than our crawler does.
As Keri and bishop23 mentioned, this is most likely not a major issue if GWT isn't reporting the errors, but we prefer to report the issues because we would rather be safe than sorry.
I hope this helps. Please let me know if you have any other questions.
Chiaryn
-
I'm not seeing an answer that jumps out at me for this one. For the immediate future, don't sweat it if you're not seeing it in GWT. This is assigned to our help desk, and we'll have someone from there investigate more and get back to you, though it might be a few days because of the Thanksgiving holiday (if you don't get an answer today, it may be Monday before we have a chance to respond).
-
If they're not appearing on WMT than you should ignore unless it's an exact duplicated content, then delete
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why does Moz see short Russian & Chinese urls as too long
We are translating content into Russian and Chinese on our website, the number of errors are increasing mainly around URL too long, each time we create a page with a Chinese or Russian url. If you click on the link below for a Chinese content page: https://www.westbourneschool.com/zh-hans/%E5%AE%BF%E8%88%8D%E5%8F%8A%E5%AF%84%E5%AE%BF%E5%AE%B6%E5%BA%AD/%E5%AE%BF%E8%88%8D%E7%94%9F%E6%B4%BB You will notice the url displayed by the browser is actually not very long, is there a way for MOZ not to see it as it appears above? Below is a page in Russian https://www.westbourneschool.com/ru/%D0%A8%D0%BA%D0%BE%D0%BB%D0%B0%20%D0%9F%D1%80%D0%BE%D0%B6%D0%B8%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5 Any help will be much appreciated.
Product Support | | mariedetitomount0 -
My site crawl has been in progress since last week
Hi there, I've been waiting on my site crawl to complete since Friday (it's Tuesday now), but it still has the 'in progress' notification at the top. Is it normal for it to take over 3 days? Or is there something holding it up?
Product Support | | VAPartners0 -
Can't reactivate account - duplicate transaction. Anyone experienced this?
as Q explains, my card expired, tried to renew subscription and get a message saying declined, duplicate transaction?
Product Support | | paulazoid0 -
Reports Issues
Hello there, I recently re-activated my account and I have some issues with the reports. I have been notified by email that the crawl has been successful and data were collected but they refer to January and February instead of November. What should I do? Thanks
Product Support | | PremioOscar0 -
On page SEO Report Sucks
Hey. I'm trying to get a PDF of the on page report so I can see a list of each issue. This works well if you're logged in and just navigate through the different tabs, but not so well if you export. All I get is an overview of the number of the priority issues, but not the URLs and the specific problems. I know you can export a CVS, but that's not a very good branded way to export this. What I'd like to do is export a report that is fairly gainular so I can provide "before and after" reports for work we completed. As is stands now, I have an overview PDF which sucks and can't provide any more details or a CVS which has all the details in the world, but is so in depth it's hard to navigate, especially for clients who don't know what they're looking at, not to mention it's not branded!!! There should be way more options for reporting here. Please let me know if you can do something about this or if I'm just missing something. Thanks. Micha
Product Support | | Multiverse-Media-Group0 -
Why can't the dates be changed in automatic reports?
I want to be able to change the publish date of automatic reports in my campaigns. One such campaign, which is a client campaign, it's set to run on the 8th after I selected "monthly". However, this doesn't work for me as this client want's to meet each month on between the 2nd and 5th of each month and I have to have this report data. So, I need to run this report on the first. Not the 2nd, not every 4 weeks... on the first. It seems like you guys have a fundamental flaw in the design of this tool, as great as it is. You've set the projects to auto run each month from the date it was added (at least from what I can tell). Provided that's true, then this would explain why the monthly reports won't work on my schedule because their on a weekly schedule instead. We, as clients, should be given the option to schedule when our scanning runs, when our reports get generated, etc. Every company runs their SEO and marketing differently, but the away you've set this up with the lack of options for us, forces us to work around your tools scheduling and not the other way around. Also, out of all of the SEO tools I've tried (quite a few), none have had this limitation. This should be addressed immediately. Thanks, Micha
Product Support | | Multiverse-Media-Group1 -
Why is my dashboard reports not updating???
Hi, Can someone from Moz Team help me out. I just checked my campaigns and they are stuck to Dec 2-8 data. It's already February LOL.
Product Support | | en-gageinc0 -
Why Isn't My Analytics Report Downloading?
In the "Preview & Download" section of reporting, I clicked the "Generate PDF Report Preview" and it initiates loading but it has been approximately 30 minutes and the report PDF still hasn't generated. Does anyone know why this may be? Thanks!
Product Support | | Danny_Laws0