Abnormal crawl issues appearing in my Moz results
-
I have been asked to look at a site for a friend and was more than surprised to see 16,9k crawl issues appear in the dashboard... of this 6,238 are duplicate page content and 5878 are duplicated page titles.
What on earth is going on? I have spoken to the web developer as it appears there is a dev site somewhere and this is his response
[Can I stress that Google determines which site was in the index first and then removes other sites it sees as having duplicate content. Our dev sites appearing in the search index would not affect your ranking due to duplicate content as Google would see your site as the first site with the content]
As I cannot make contact with him, I am scratching my head, surely a dev site should be no-indexed, it sounds as though he is saying that its ok because Google will take the main site as the first site with the content...
Very confused! Help need MOZ community.
Manythanks,
Sarah
-
Thanks again Dirk. I like your direct and knowledgeable responses. I have sent a Linkedin connection!!
Many thanks,
Sarah
-
Hi Sarah,
Googlebot will follow these links as well and discover these "useless" pages (the are off course not useless from human perspective but they don't add value for bots - and they will be considered as duplicates). Duplicates are no reason for "punishment" - so you could just let them be. Personally I would put a nofollow on these links or add a "noindex" tag to the login page. Normally you shouldn't use nofollow on internal links - but login pages are an exemption on this (check also https://searchenginewatch.com/sew/news/2298312/matt-cutts-you-dont-have-to-nofollow-internal-links : "Of course, there are always exceptions to the rule, and things like login pages can be the exception. He said it doesn’t hurt to put the nofollow link for a link pointing to a login page, or things like terms and conditions or other “useless” pages. However, it doesn’t hurt at all for those pages to be crawled by Google."
For the practical part - if you add an additional question to a question which has been marked as answered - only the ones who have already answered will see the additional question. To be on the safe side - it's better open a new question if you want other people to have a look at it.
Hope this helps,
Dirk
-
hello Dirk, thank you for that great answer, we have since been doing a bit more digging of our own and before we go back to the web developer we want to check what should be happening with the links the we are finding duplicated as we are seeing that the issues relating to Duplicate Pages are coming from links from the login page which shows information about where the user was redirected from.
For example, if the visitor is not logged on and wishes to wish-list an item, they will be redirected to the login page, with the item code and intended action in the url; which can then continue on to the desired page once logged on.
The MOZ crawler is seeing these pages as having Duplicated Content whilst they are all the same apart from a piece of information in the URL. Should we be blocking these duplications? Are they a risk to us? What should we be doing?
I have also added this as a new question - I am quite new to this community thing so wasn't sure which was the best way to ask the question.
Many thanks again,
Sarah
-
Moz is only indexing pages it's crawler is able to find. This implies that on your production site you have links to your development site.
Don't really agree with what your dev is saying - he should correct these links first; put a noindex on these pages. Alternative - put a password on the dev site so it's only accessible with a password. If a lot of users are putting links to your dev site it could become more important than your main site. Google will try to choose the most appropriate site - but you have no guarantee that it will choose the right version. In any case - that's not the type of risk you should be willing to take.
Once this is done - you can request a removal of these pages via the search console.
If all pages are removed from the index you can adapt the robots.txt to prevent access to the Google & other bots. Do this only after all pages are removed - if not Google will never find the noindex directive.
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Pro
Anyone felt that moz pro isn't accurate like in search volume of products and links data that its slow in it?
Moz Pro | | apcreativity0 -
Malware on Moz?
Hi guys. Today at evening I received few notifications from my anitivirus about malware on Moz. Does anyone have similar issue? q8gWqIo.jpg
Moz Pro | | Dmitriy_S0 -
Duplicate content in crawl despite canonical
Hi! I've had a bunch of duplicate content issues come up in a crawl, but a lot of them seem to have canonical tags implemented correctly. For example: http://www.alwayshobbies.com/brands/aztec-imports/-catg=Fireplaces http://www.alwayshobbies.com/brands/aztec-imports/-catg=Nursery http://www.alwayshobbies.com/brands/aztec-imports/-catg=Turntables http://www.alwayshobbies.com/brands/aztec-imports/-catg=Turntables?page=0 Aztec http://www.alwayshobbies.com/brands/aztec-imports/-catg=Turntables?page=1 Any ideas on what's happening here?
Moz Pro | | neooptic0 -
On page analysis showing old results
Hi, My on page crawl analysis is showing that I have 38 duplicate content issues. These issues are because we had used tags in our blog. We remove all the tags 10+ day ago but the crawl that was done 5 days ago is still showing the tags as causing a duplicate content issue. Why would this be? We cannot correct is as it is already corrected. How do we get moz to crawl the correctly? Thanks Andrew
Moz Pro | | Studio330 -
When I did my first crawl, I was given some errors.
Do I then need to re-crawl to make sure the errors were fixed accordingly?
Moz Pro | | immortalgamer0 -
Crawl Test produced only 1 page
Hi, I recently submitted a crawl for www.cirrato.com using SEOMoz Crawl Test Tool. I have a lot of pages, but the crawl result shows only 1 page, which is the front page and nothing else... Does anyone know what this could mean or what the problem is?
Moz Pro | | yusufcirrato0 -
Changing the Timeframe of Historical Crawl Data
Hello, Just read a great post about the implications of duplicate content for sites after the most recent Panda update: http://www.seomoz.org/blog/duplicate-content-in-a-post-panda-world?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+seomoz+(SEOmoz+Daily+Blog) In the post is an image or crawl data history that shows months, not days or weeks, worth of trending data as it relates to duplicate content. So my question is this: How do I change my view/date range on my own campaigns so that I can view the trailing months of data rather than only what seems to be the past 4 weeks or so? This would really help me identify the impact of some on page changes we've recently made for a client. Many Thanks, Jared
Moz Pro | | surjm0 -
API returns only 3 results
Hi all, I'm using the links seomoz API, and it's working great for me, except it only appears to return 3 results. Seems to only return 3 results on the anchor text api also. Is there something basic I'm not doing to get more results? Thanks!
Moz Pro | | jimbo_kemp0