How can I find duplicate pages from a Moz Crawl?
-
We have many duplicate pages that show up on the Moz Crawl, and we're trying to fix these but it's very difficult because I can't see a way to isolate the code where the duplicate is found. For instance, http://experiencemission.org/immersion/ is one of our main pages, and the crawl shows one duplicate of http://experiencemission.org/immersion. It appears that one of our staff manually edited the source code in one of our pages but forgot the trailing slash. This would be an easy fix but the problem is that this page is linked to internally on our website 2423 times, so it's next to impossible to find the code that is incorrect. We have many other pages with this same basic problem. We know we have duplicates, but it's next to impossible to isolate them.
So my question is this: When viewing the Moz Crawl data is there any way to see where a specific duplicate page link is located on our website?
Thanks for any and all help!
-
Thanks for taking the time to respond. The open site explorer is helpful for issues that have a manageable number of internal links. However, for the example above and a few others like it on our website it is not that helpful because isolating the link would still require us to click on the pages individually to view the source code. This is because most of our errors are minor errors such as an omitted slash or capitalization. Such errors are flagged as duplicate content in our Moz crawl but the links still work because they redirect to the correct page and thus they are not able to be isolated on the open site explorer. Unfortunately the .csv is no help at all because it only shows the page being linked to not the page where the actual link is coming from.
Are we just out of luck on this or is there another option?
-
Hey there! You've got a couple different options for ways to track this information down. The first would be to head into your campaign, head over to the Site Crawl and click on the link towards the bottom for Duplicate Page Content. Right below the graph you'll see a button that says Download CSV. Open that up and head on over to column AM and you'll see the referring URL! Another option is to jump into Open Site Explorer and check out the internal inbound links. Hope this helps and let us know if you need anything else!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Crawl only crawling the top level page (1 page)
For the past few mounts my weekly site crawl has been inconsistent. One week works fine, it crawls all of my 500 or so pages. The following week it only crawls 1 page (http://mydomain.com) and nothing else. A few weekly scan go by and the crawl is back up the the 500 or so pages.I went ahead and created several campaigns with duplicate settings and crawled the site. Most times but not all the new campaign's crawl works fine crawling all pages. But within a week or two the weekly crawl will fail again. (crawling 1 page). Currently i have four campaign's all with the same settings running weekly crawls. 2 campaign's crawled the 500 pages and two crawled only the single page. Any help will be greatly appreciated
Moz Bar | | dmaude0 -
Duplicate content found in scan
On June 8th we ran a Moz Crawl on our site. We found 144 pages that were flagged with duplicate content.
Moz Bar | | StickyLife
Again on June 13th we ran another moz crawl on our site and found 137 pages that were flagged with duplicate content. Then one final scan on June 22nd with 161 pages of duplicate content. After comparing the 3 different scans I see that, without making any changes, pages that were not flagged as duplicate content are now being flagged as duplicate content. While at the same time, pages that were originally flagged as duplicate content are now no longer showing up with duplicate content. I could understand if we made some changes to these pages but no changes were made. For example: On the 8th this page was flagged as duplicate content - https://www.stickylife.com/star-magnet
On the 13th and 22nd it was not flagged as duplicate content but no changes were made to that page. For reference it was flagged as duplicate content with the following page: https://www.stickylife.com/baseball-glove-magnet This page was also Not changed or altered between between these dates. In addition, when Moz scans our site through our campaign every Friday the results do not match what we see when we do a manual scan. Moz's weekly scan only reveals 14 pages with duplicate content as opposed to the numbers you see above. Why such inconsistencies in the Moz Scans?0 -
Why RogerBot can't crawl site https://unplag.com
Hello Please help me to solve the problem. The on-page grader and Crawl Test are not working for Unplag.com website. Both said that they can't access the url. Yes, I've tried different variants like unplag.com, http://unplag.com One more thing - RogerBot was disallowed in robots.txt file. I deleted it from the file a week ago so maybe moz index haven't been renewed.
Moz Bar | | Targeras0 -
Moz Crawler URL paramaters & duplicate content
Hi all, this is my first post on Moz Q&A 🙂 Questions: Does the Moz Crawler take into account rel="canonical" for search results pages with sorting / filtering URL parameters? How much time does it take for an issue to disappear from the issues list after it's been corrected? Does it come op in the next weekly report? I'm asking because the crawler is reporting 50k+ pages crawled, when in reality, this number should be closer to 1000. All pages with query parameters have the correct canonical tag pointing to the root URL, so I'm wondering whether I need to noindex the other pages for the crawler to report correct data?: Original (canonical URL): DOMAIN.COM/charters/search/mx/BS?search_location=cabo-san-lucas Filter active URL: DOMAIN.COM/charters/search/mx/BS?search_location=cabo-san-lucas&booking_date=&booking_days=1&booking_persons=1&priceFilter%5B%5D=0%2C500&includedPriceFilter%5B%5D=drinks-soft Also, if noindex is the only solution, will it impact the ranking of the pages involved? Note: Google and Bing are semi-successful in reporting index page count, each reporting around 2.5k result pages when using the site:DOMAIN.com query. The rel canonical tag was missing for a short period of time about 4 weeks ago, but since fixing the issue these pages still haven't been deindexed. Appreciate any suggestions regarding Moz Crawler & Google / Bing index count!
Moz Bar | | Vukan_Simic0 -
MoZ vs Alexa & Moz vs Google
My colleague is continuously arguing with me why i went for Moz and why not for Alexa, He also says when Google is there then why Moz. I tried on my part to convince him but he has his own learning. Can anybody help me make him understand otherwise my job will become hard if he remains doubtful about Moz? Looking for down to earth and an honest feedback. Thanks Tanveer
Moz Bar | | Sequelmed0 -
Can't delete items from the on page grader
I check every single box and they don't delete. This is driving me nuts. Please can you delete them for me because I am not impressed with this AT ALL. In fact I am getting so cross I am in danger of screaming hysterically which might get me the sack and it would be all your fault. That was slightly tongue in cheek, but please can you fix it please. please.
Moz Bar | | CommT0 -
Monthly Data Option not appearing in Moz AGAIN
Hello Again, It's a real shame that I seem to come to this forum to only complain about your product again and again. I remember a time when I used to really respect this company, but your tool is absolutely abysmal. Anyway, my latest issue (to add to the pile) is that campaigns that have been in my account for almost 2 months STILL do have options for me to make monthly reports on. What's wrong now? Edit: I just checked the date I added these profiles, it was around the end of last month, so one and a half months ago.
Moz Bar | | Paul_Tovey0 -
Emails from Moz makes my Outlook unresponsive
Did anybody else notice this? It started a few weeks ago, every time that I receive an email from Moz regarding a Q&.A update and I try to open it, my Outlook becomes unresponsive and I have to restart it.
Moz Bar | | echo10
