Moz Crawler suddenly reporting 1000s of duplicates (BE.net)
-
In the last 3-4 days we've had several thousand 'duplicate content' warnings appear in our crawl report, 99% of them related to our on-site blog. The blog is BlogEngine.Net, but the pages simply don't exist. The majority seem to be Roger trying quasi-random URLs like:
/?page=410/?page=151
Etc. etc. The blog will present content for these requests, but it is of course the same empty page since there's only unique content for up to /?Page=10 or so.
Two questions:
1. Did something change recently? These blogs have been up for months, and this problem has only come up this week. Did Roger change to become more aggressive lately?
2. Suggested remediation? On one of the blogs I've put no-index no-follow for any page that has a /?page querystring, and we'll see what effect that has come next crawl next week. However, I'm not sure this will work as per:
http://moz.com/community/q/functionality-of-seomoz-crawl-page-reports
Anyone else had dynamic blogs suddenly blossom into thousands of duplicate content warnings? Google (rightly) ignores these pages completely.
-
Hate to bump my own question, but it appears I spoke too soon about no-index,no-follow solving this. The duplicate errors went away for about 5 days, but then yesterday spiked with the same problem. I've confirmed that no-index, no-follow are present on the pages being detected as bad.
As per the best practices document:
http://moz.com/learn/seo/robotstxt
Using meta robots no index no follow is the recommended option:
Block with Meta NoIndex
This tells engines they can visit, but are not allowed to display the URL in results. This is the recommended method
But it apparently isn't working, as evidenced by the new surge of duplicate errors. Is there anything else I can do? I don't want to explicitly block Roger in robots.txt as that seems rather backward. Should Roger be included the Bad Robots List?
-
Peter -
Thanks for the clarification. I understand the philosophy at hand, and I kind of even understood it before I had asked the question. I'm handling these with a mix of canonical and no-index/no-robot.
Related to that, update:
By marking the superfluous pages no-index/no-follow the error count for the site has diminished by about 10,000 and the warning count by about 28,000 so that seems to be the way to go. The pages that had content are 'low value' in this context, since that content was readily available elsewhere.
-
Hi there!
Thanks for writing in with a great question.
We definitely count those dynamic URLs as duplicate content. While we are pretty sure that search engines can figure this stuff out and know which URL to index, it's still considered best practices to canonicalize or otherwise direct crawlers to the original URL (as far as I know. I'm not a professional SEO so you might be better off asking the Pro Q&A community at www.moz.com/community/q - they are all SEOs like you).
Since some dynamic URL generators can cause problems for crawlers, we do try to be overly-inclusive of these issues rather than overly-exclusive. We want people to know about potential issues with sites, even if they're not really issues in the scheme of the site owner's specific SEO implementation plan.
In sum, we'd rather leave those judgments up to you and at the same time, provide you with the data you need to make these decisions. I hope this helps explain our thinking here! However, if you think that our crawler might be having issues, and you do not want to post your site urls here you could always send us a support ticket at help@moz.com. That way can can examine it a bit further and provide some insights into why our crawler thinks this way!
Hope this helps!
Peter
Moz Help Team.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I am confuse with google analytic custom and segment report
Hi All, In google analytic when I create custom report for my ecommerce site then figures go mad. I really not able to judge peformance of device, browser and it's version, conversion, ecommerce conversion rate etc. same way if I add secondary dimension in report then also figures are not accurate. Again when I create different segment like desktop, mobile, tablet then in tablet segment mobile devices comes and in mobile device tablet appear why segment also not accurate? Is it because I am using free version? Also do we have alternative of google analytic which give same report like google analytic either device, browser, os, segment, enhance ecommerce etc? Thanks!
Reporting & Analytics | | dhisman0 -
Duplicate meta descriptions
Hi all,
Reporting & Analytics | | Roi_Bar
Google webmaster point me to duplicate meta descriptions problem between pages with difference URL but have the same canonical URL.
What could be the reason? https://treato.com/ParaGard,tired/?a=s
https://treato.com/ParaGard,Tiredness/?a=s
https://treato.com/ParaGard,tired/?a=s
https://treato.com/ParaGard,Tiredness/?a=s gKUSXsU1 -
Why would page views per visitor suddenly increase?
My website traffic is growing by about 1% a week. It has a fairly stable page views/visitor of about 1.69. There's normally very little variability in this As we sell an industrial product. Today page views jumped by 50% and so did page views/visitor but visitor numbers stayed the same. I dont have a useful hypothesis to explain this. Analytics shows me that the traffic source, country of origin and pages viewed are pretty much the same as normal. There's been no substantive change to the site (today we changed the text in a widget to link to a new page - and no one visited it). It doesn't look like 1 person has gone through the whole site as that would skew the distribution of page views by country So why would user behavour suddenly change? I'll look at it for the rest of the week but in 7 years of looking after this website I haven't seen anything like this before.
Reporting & Analytics | | Zippy-Bungle0 -
Google webmaster links vs Moz Reporte do follow links
A bit confused about my seo reports for a site I am tracking in Moz. Google webmaster reports i have 1836 links to my domain.
Reporting & Analytics | | KenW
Moz reports 273 external followed links.
Website Auditor reports 449 dofollow and 338 no follow.> total 787
What is important factor that I should be reporting to my client that really matters?0 -
Why do I have differences in my User Flow Report?
On the user flow report I am showing a first interaction of /category as my second highest starting page. If I click group details it shows some of my different categories. However, at the bottom of the user flow report it has some of the category pages out as individual categories. Why is this? And how do I force it to show all category pages together like it does with my /item pages?
Reporting & Analytics | | EcommerceSite0 -
When i first add my url to seomoz then i had a general report of all the faults my website had in SEO and suggestions where can i find it now , i cant find it ?!
when i first add my url to seomoz then i had a general report of all the faults my website had in SEO and suggestions where can i find it now , i cant find it ?!
Reporting & Analytics | | fireproductsuk0 -
Setting up GA to report on sub domains
I've just launched a mobile version of my site at m.mydomain.com. I could do with a bit of help just checking I've inserted my Google Analytics code correctly though. I've chosen the sub domain option from settings and pasted the code into both the mobile and the non-mobile version of my sites. However, am I right in thinking that the setDomain attribute should point to two different domains for example: _gaq.push(['_setDomainName', 'm.mydomain.com']); (for the mobile site) and: _gaq.push(['_setDomainName', 'mydomain.com']); (for the standard site) Thanks very much for your help!
Reporting & Analytics | | theshortstack0 -
On what report do I get to know where do the external links come from?
I need to get a list of the external links. I don't find that report, where do I get it?
Reporting & Analytics | | carloscontinua0