Welcome to the Q&A Forum

We're measuring using a few different tools. Right now if I'm looking with the Google Preview Tool, it can sometimes show me what the deal is. I'm sure it's not the most accurate, but a ranking drop of 10 pages isn't hard to measure :P.

If there hasn't been any sort of update, I'm pretty sure I know what the problems are otherwise. I think they may have just updated a disavow file for us.

Today, the ranking is slowly crawling back up the list.

Thanks for the response, Eric. When these major updates, or even smaller updates happen, do you commonly see an up and down  shift for a single page? Do they fluctuate and then settle in a spot, or is it usually a one way shift for pages?

Has anyone else checked their rankings today and noticed a big shift? We just got smacked by G big time.

Hate to bump my own question, but it appears I spoke too soon about no-index,no-follow solving this. The duplicate errors went away for about 5 days, but then yesterday spiked with the same problem. I've confirmed that no-index, no-follow are present on the pages being detected as bad.

As per the best practices document:

http://moz.com/learn/seo/robotstxt

Using meta robots no index no follow is the recommended option:

Block with Meta NoIndex

This tells engines they can visit, but are not allowed to display the URL in results. This is the recommended method

But it apparently isn't working, as evidenced by the new surge of duplicate errors. Is there anything else I can do? I don't want to explicitly block Roger in robots.txt as that seems rather backward. Should Roger be included the Bad Robots List?

Peter -

Thanks for the clarification. I understand the philosophy at hand, and I kind of even understood it before I had asked the question. I'm handling these with a mix of canonical and no-index/no-robot.

Related to that, update:

By marking the superfluous pages no-index/no-follow the error count for the site has diminished by about 10,000 and the warning count by about 28,000 so that seems to be the way to go. The pages that had content are 'low value' in this context, since that content was readily available elsewhere.

In the last 3-4 days we've had several thousand 'duplicate content' warnings appear in our crawl report, 99% of them related to our on-site blog. The blog is BlogEngine.Net, but the pages simply don't exist. The majority seem to be Roger trying quasi-random URLs like:
/?page=410

/?page=151

Etc. etc. The blog will present content for these requests, but it is of course the same empty page since there's only unique content for up to /?Page=10 or so.

Two questions:

1. Did something change recently? These blogs have been up for months, and this problem has only come up this week. Did Roger change to become more aggressive lately?

2. Suggested remediation? On one of the blogs I've put no-index no-follow for any page that has a /?page querystring, and we'll see what effect that has come next crawl next week. However, I'm not sure this will work as per:

http://moz.com/community/q/functionality-of-seomoz-crawl-page-reports

Anyone else had dynamic blogs suddenly blossom into thousands of duplicate content warnings? Google (rightly) ignores these pages completely.