Duplicate Content Report: Duplicate URLs being crawled with "++" at the end
-
Hi,
In our Moz report over the past few weeks I've noticed some duplicate URLs appearing like the following:
Original (valid) URL:
http://www.paperstone.co.uk/cat_553-616_Office-Pins-Clips-and-Bands.aspx?filter_colour=Green
Duplicate URL:
http://www.paperstone.co.uk/cat_553-616_Office-Pins-Clips-and-Bands.aspx?filter_colour=Green**++**
These aren't appearing in Webmaster Tools, or in a Screaming Frog crawl of our site so I'm wondering if this is a bug with the Moz crawler? I realise that it could be resolved using a canonical reference, or performing a 301 from the duplicate to the canonical URL but I'd like to find out what's causing it and whether anyone else was experiencing the same problem.
Thanks,
George
-
So glad to help, George!
-
Hi Chiaryn,
Thanks - you've been really helpful! I had assumed that as the referrer wasn't in the Web UI (per WMT), it wasn't available anywhere. I'd also assumed it was a copywriting issue and not a product data issue.
Need to readdress my assumptions
George
-
Hey George,
Thanks for writing in.
I looked into the pages with the ++ in the URL and it seems that they do actually exist on the site, so it isn't an issue with our crawler that is causing these in your crawl errors. For example, a link to the URL http://www.paperstone.co.uk/cat_553_Desktop-Essentials.aspx?filter_colour=Green++ can be found in the source code of the page http://www.paperstone.co.uk/cat_553_Desktop-Essentials.aspx here: http://screencast.com/t/HpHTlSs5gH8H
You can find the referral pages for the ++ pages on the site by downloading the Full Crawl Diagnostics CSV. In the first column, perform a search for the ++. When you find the correct row, look in the column labeled referrer, AM. This tells you the referral URL of the page where our crawlers first found the URLs that include ++. You can then visit this URL to find the links to those pages.
Since these URLs with the ++ do resolve with a 200 http status and they have the same code and content as the pages without the ++, our crawler will count them as duplicate content. I'm not certain why Screaming Frog and GWT are not find or reporting these pages; it may be that they parse the + signs in the URL differently than our crawler does.
As Keri and bishop23 mentioned, this is most likely not a major issue if GWT isn't reporting the errors, but we prefer to report the issues because we would rather be safe than sorry.
I hope this helps. Please let me know if you have any other questions.
Chiaryn
-
I'm not seeing an answer that jumps out at me for this one. For the immediate future, don't sweat it if you're not seeing it in GWT. This is assigned to our help desk, and we'll have someone from there investigate more and get back to you, though it might be a few days because of the Thanksgiving holiday (if you don't get an answer today, it may be Monday before we have a chance to respond).
-
If they're not appearing on WMT than you should ignore unless it's an exact duplicated content, then delete
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Solved Why is MOZ crawl taking so long?
I began my site crawl on November 3rd and now it is November 7th and it is still "in progress". Why is this happening?
Product Support | | CarisaS_Wenda0 -
Emails from Moz display incorrect report information
Hey, I have noticed that for the last 3 weeks or so the emails from Moz have been displaying incorrect information. The results it shows in the email seem to be from previous crawls. For example The previous week a sitemap that didn't update that caused 400+ pages to appear as 404's. This wasn't reported in the email and it stated there were only 2 errors on the site. (Of which we knew about anyway and choose to ignore). Thankfully i checked the crawl report anyway out of habit and saw the onsite report to show 400+ crawl errors. I went on to fix these errors. This week, the crawl report showed 400+ crawl errors, I immediately logged on to the website to check the campaign to find there are only the 2 expected crawl errors. It is almost as if the info from the campaign is being pulled and emailed to me seconds before it is updated from the crawl info. I am finding this to be the case in both the keyword and crawl errors email reports.
Product Support | | ATP0 -
Older Reports
I'm newer to Moz...Is there a way for me to look at older reports? For example, I want to compare January numbers with December numbers but I did not save my December reports. I'm hoping this is just something small, but please let me know if this is possible!
Product Support | | Monitronics20150 -
Where are my October Monthly Reports?
My custom monthly reports have not been generated for any of our clients for the month of October. The report lists 10/1/2013 as the last report and says that the next report will be available on 12/3/2013. Obviously, there is a glitch that has caused October to be skipped. How can I fix this in order to get my monthly reports to my clients ASAP?
Product Support | | accpar0 -
Report is scheduled for next month - need it now
Hi all, I just made a report from Moz pro. After selecting all the reports I wanted clicked finished. My report would send by e-mail to me and my account address. But I still haven't received any report.
Product Support | | StercBV
When I check the 'manage report' page it says that my next report will be created in july. But I need it now 🙂 It has been about 2 hours now since I created the report. Can someone tell my when I've to be more patience or it will take up to july for my first report? My campaign allready is added more then a month ago. First time to export the reports as complete set0 -
Traffic Data Report Error: Anyone else?
I included a screenshot of the report. I had a 111 organic visits for last Mondays past week report, this Monday it is telling me I had 0. It's also telling me I have -10 non-branded keywords. I know the organic search visits isn't 0. Also, (this part I'm not sure about) but I don't see how I could have a negative amount of non-branded keywords. I would think it would just stop at 0. Has anyone had something like this happen? Or am I just missing something? Thanks for any help or comments. zK3FbG4.png
Product Support | | KempRugeLawGroup0