This is a good question. I think "spammy" needs to be defined since sites like ours, up for many years, have links in numerous places that cannot be "fixed" because at the time there was no such thing s "nofollow" and the like. To punish for old links would seem harsh although it might be done. Our problem is that we're listed on long pages of "link resources" that many are saying is spam and penalty but this is what was done in the 90s and 2000s by people wanting to put up useful websites. Here are a few I saw on possibly relevant articles that appear knowledgeable:
- http://searchenginewatch.com/article/2271305/Google-Penguin-2.0-Casualties-Why-Sites-Got-Hit
- http://www.searchenginejournal.com/googles-penguin-beginning-to-flex-its-muscles/63416/
I find Barry's articles very helpful and ostensibly honest: http://searchengineland.com/googles-matt-cutts-on-upcoming-penguin-panda-link-networks-updates-151273
To begin, it's difficult for anyone to tell you what the problem is with your site without providing your site. It's a horrible feeling (I know) but that will give you the best answers, unless you prefer to pay for that privacy by hiring an SEO consultant to manage the issue. You may want to post some of the links here if you don't want to post your site and see what the group says. Regarding blogrolls - another issue that affects the 2000s and it's crazy for Google to think we can "clean" that which never really needed cleaning and only the large companies with staff will survive. Hopefully there will be clarity. Best of luck.
PS - If it makes you feel any better, your traffic drops are small in comparison to some others.