if all pages are 100#5 not indexed then I would block it in robots.txt, Google's John Muller confirmed to me that Googlebot will continue to crawl every link to check to see if a nofollow or noindex has changed status.

So as a result we blocked our pages with robots.txt and saw a great increases in index/crawl rates on pages we want Google to pay attention to. It also reduces waste in server resources.

However if there are any pages that are index, if you block them in robots.txt then Googlebot will never be able to crawl the link to determine that it should be noindex. This means it could stay in a permanent stage of indexed.

I hope that answers all your questions?

This is a Panda issue.

Google has said many times with affiliate sites that use the same content that if they do a better job than the original site it will rank them. So its not all bad when you look at it from that point of view.

However, Google loves unique content and will do its best to rank sites first that have the unique content. I have a business in the office space industry and a few years back we used to aggregate office apace listings which were shared amongst 30+ sites. The display of these listings would be different for many searches but the content was the same as all the other sites. This slowly put us in a PANDA DEVALUATION (there is no panda penalty).

After re-writing them with our clients we saw a significant change once the content had be re-crawled.

So it can have a great effect. If Google starts to see that large parts of your site are duplicate content it will start to question the authority you have in your industry.

Could you offer and incentive to your customers to write something unique? And also maybe inform your users not to copy and paste their own content on your site as this could affect them negatively in Google?

If you are an authority could you tell users that if you want to be listed it must be unique? Or if its a paid service have an ad on service for a few bucks where you write a professional description? Might become a nice additional income?

Just a few ideas

We decided that it was worth a large investment as we would own the content ourselves and not worry in the future about anyone claiming ownership to the content as google gets stricter. So we re wrote half a million words!

A penalty is something google will have to manually remove and you will be able to see that in webmaster tools. A devaluation is when you are adjusted by the algorithm and lowered as a result because each thing that google does not like acts as points against you but you can quickly change and see your results return. Does that make sense?

Yes, actually John Muller at Google said this is a wise thing to do.

First ask yourself is there really any benefit in your users going to any of those pages?

Duplicate content is not a major issue and can be wildly ignored but many blogs go over the top with tags etc.. Making Google's life harder when trying to make a decision on what content to show for results.

I personally went as far as removing the front end of wordpress completely and coded a very quick and simple version of the front end myself, so it still gets managed using the wordpress interface like a content management system.

Works a treat and the response from Google SERPS was great. Also reduces the amount of pages Google needs to crawl over and over to look for changes. This will see Google returning more frequently to the key pages looking for updates.

Question is do your customers pay to be listed with you? If so are they using you for the dofollow links? If this is the case then you may lose some of your business by changing it to nofollow.

If they are paying there is also a risk of a Google penalty for paid do follow links.

If you are unable to maintain quality and there is no good reason to have a dofollow, then switch to nofollow.

Are the pages hardcoded? Or is all the data in a database? If it is in a database it would take no time at all to run each domain through a loop and check what response status code you get.http://en.wikipedia.org/wiki/List_of_HTTP_status_codes
This would be a very quick way to find broken links. You may even be able to purchase an api on something like majestic or Moz and run the sites through that as well for a better indication of site quality. If the site has very low DA or Trust Flow, you could also make it nofollow or remove etc...

If it is all hardcoded then that would be very hard work all around.