Abnormal crawl issues appearing in my Moz results
-
I have been asked to look at a site for a friend and was more than surprised to see 16,9k crawl issues appear in the dashboard... of this 6,238 are duplicate page content and 5878 are duplicated page titles.
What on earth is going on? I have spoken to the web developer as it appears there is a dev site somewhere and this is his response
[Can I stress that Google determines which site was in the index first and then removes other sites it sees as having duplicate content. Our dev sites appearing in the search index would not affect your ranking due to duplicate content as Google would see your site as the first site with the content]
As I cannot make contact with him, I am scratching my head, surely a dev site should be no-indexed, it sounds as though he is saying that its ok because Google will take the main site as the first site with the content...
Very confused! Help need MOZ community.
Manythanks,
Sarah
-
Thanks again Dirk. I like your direct and knowledgeable responses. I have sent a Linkedin connection!!
Many thanks,
Sarah
-
Hi Sarah,
Googlebot will follow these links as well and discover these "useless" pages (the are off course not useless from human perspective but they don't add value for bots - and they will be considered as duplicates). Duplicates are no reason for "punishment" - so you could just let them be. Personally I would put a nofollow on these links or add a "noindex" tag to the login page. Normally you shouldn't use nofollow on internal links - but login pages are an exemption on this (check also https://searchenginewatch.com/sew/news/2298312/matt-cutts-you-dont-have-to-nofollow-internal-links : "Of course, there are always exceptions to the rule, and things like login pages can be the exception. He said it doesn’t hurt to put the nofollow link for a link pointing to a login page, or things like terms and conditions or other “useless” pages. However, it doesn’t hurt at all for those pages to be crawled by Google."
For the practical part - if you add an additional question to a question which has been marked as answered - only the ones who have already answered will see the additional question. To be on the safe side - it's better open a new question if you want other people to have a look at it.
Hope this helps,
Dirk
-
hello Dirk, thank you for that great answer, we have since been doing a bit more digging of our own and before we go back to the web developer we want to check what should be happening with the links the we are finding duplicated as we are seeing that the issues relating to Duplicate Pages are coming from links from the login page which shows information about where the user was redirected from.
For example, if the visitor is not logged on and wishes to wish-list an item, they will be redirected to the login page, with the item code and intended action in the url; which can then continue on to the desired page once logged on.
The MOZ crawler is seeing these pages as having Duplicated Content whilst they are all the same apart from a piece of information in the URL. Should we be blocking these duplications? Are they a risk to us? What should we be doing?
I have also added this as a new question - I am quite new to this community thing so wasn't sure which was the best way to ask the question.
Many thanks again,
Sarah
-
Moz is only indexing pages it's crawler is able to find. This implies that on your production site you have links to your development site.
Don't really agree with what your dev is saying - he should correct these links first; put a noindex on these pages. Alternative - put a password on the dev site so it's only accessible with a password. If a lot of users are putting links to your dev site it could become more important than your main site. Google will try to choose the most appropriate site - but you have no guarantee that it will choose the right version. In any case - that's not the type of risk you should be willing to take.
Once this is done - you can request a removal of these pages via the search console.
If all pages are removed from the index you can adapt the robots.txt to prevent access to the Google & other bots. Do this only after all pages are removed - if not Google will never find the noindex directive.
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Does Moz Pro include Moz Local
My client has bought about six Moz Local accounts and are pleased with results. We have not yet used your Moz Pro program. The client might be interested in switching to the Moz Pro if those Moz Local accounts can be included into it. Please let me know as soon as possible. Thanks!
Moz Pro | | gallowaywebteam0 -
Woocommerce filter urls showing in crawl results, but not indexed?
I'm getting 100's of Duplicate Content warnings for a Woocommerce store I have. The urls are
Moz Pro | | JustinMurray
etc These don't seem to be indexed in google, and the canonical is for the shop base url. These seem to be simply urls generated by Woocommerce filters. Is this simply a false alarm from Moz crawl?0 -
Crawl Diagnostics 2261 Issues with Our Blog
I just recently signed up for MOZ, so much information. I've done the walk through and will continue learning how to us the tools. But I need your help. Our first moz crawl indicated 2261 issues (447 404's, 803 duplicate content, 11 502's, etc). I've reviewed all of the crawls issues and they are linked to our Yahoo hosted WordPress blog. Our blog is over 9 years old. The only issue that I'm able to find is our categories are not set up correctly. I've searched for WordPress assistance on this topic and cant find any issues with our current category set up. Every category link that I click returns Nothing Found Apologies, but no results were found for the requested archive. Perhaps searching will help find a related post. http://site.labellaflorachildrensboutique.com/blog/ Any assistance is greatly appreciated.
Moz Pro | | Girlstuff0 -
Can Moz tools help me with this effort?
I want to brainstorm different keywords, look for high search volumes, and low competition. Then I want to create landing page, rank them using seo techniques, and collect optin email addresses so I can communicate with interested users and build helpful products. With all the moz tools available to us, how can I accomplish the above mentioned goal? Have you done some thing similar? What are your experiences? Am I talking pie in the sky? Are there any practical examples where all these steps were executed? Thanks
Moz Pro | | zsyed0 -
Ignore Moz Notices?
Hi All, I read an old post (2011) about ignoring rel canonical notices from Moz. I wondered why they can be ignored? The thing is, it's showing a bunch of rel canonical errors for my pages but I know I have the tag right. At the risk of being offensive.. can the notices be ignored because the Moz data is erroneous? I'm not sure what to make of this. Thanks!
Moz Pro | | xvpn9020 -
Issue: Title Element Too Long !
hello , i found this issue in seomoz campaigns. i see many links blocked by google for this reason . but something wrong here for example i see this link like that in seomoz dashboard Wagdy Hassan, Author at Seo Seek : Seo Tutorials , Seo Tools , Make Money Online http://seo-seek.com/author/wagdys/ and its already not in google. but when i open the link http://seo-seek.com/author/wagdys/ the title is " Wagdy Hassan " i fixed it 6 days ago. what is wrong with the site? also i still waiting google to put the new results .. wait for your answer, Thanks 🙂
Moz Pro | | Wagdys0 -
SEO Moz Rank Report
Anyone else noticing the rank reporting seems to be off? My reports are showing "not in top 50" for some keywords that my site is on page one for. Not in places listings either because I know that can throw things off! Anyone?
Moz Pro | | RickyShockley0