Hello CalicoKitty,
I've been dealing with something somewhat similar for my website (slight drop in rankings, but in my case it includes a targeted penalty, not sitewide thank goodness!) If I were you I'd definitely look at your backlinks. I've disavowed and asked for a bunch of mine removed.
Since you don't have a penalty but are losing some rankings, here are some references that have helped me decide which links are worth keeping (along with everything else that goes along with trying to get rid of a penalty).
The quick and dirty rule of thumb I use (in addition to using the Moz Bar to also gauge the value of a link) is do I think this website is genuinely built for real people who might find what I'm talking about or selling useful. If it's on the fence, I generally leave it (especially since you don't have a current penalty) but if you know in your gut it doesn't seem helpful at all, and is built more for the search engines and looks like a junk page, then you're probably looking at a bad link.Identifying Link Penalties, 2012: http://moz.com/blog/identifying-link-penalties-in-2012He does a nice job of comparing a good link profile with one that has a lot of bad links. (Essentially the same anchor text.)You can find a lot of good articles on the subject by Mozer Marie Hayne. She also wrote an excellent book on the topic available here. http://www.unnaturallinksrecovery.com. It walks you through every kind of link you might run across. The ebook is about $100 though, so I personally wouldn't buy it if I don't have a penalty on my website. That said, I bought it and I'm not sure how I would have ever worked through my penalty without her step-by-step assistance in the book.http://moz.com/community/users/308135In depth look at quality guidelines.http://moz.com/ugc/what-is-an-unnatural-link-an-in-depth-look-at-the-google-quality-guidelinesMy answered question on which links to remove, which Marie weighed in on:http://moz.com/community/q/manual-action-when-requesting-links-be-removed-how-important-to-google-is-the-address-you-re-sending-the-requests-from#reply_256531How to conduct a link audit:http://searchenginewatch.com/article/2207168/How-to-Conduct-a-Link-Audithttp://searchenginewatch.com/article/2330944/7-Things-You-May-Not-Know-About-Googles-Disavow-Tool - Mariehttp://moz.com/community/q/is-this-negative-seo-should-i-disavow-these-links
https://www.youtube.com/watch?v=K7WK3s693oI&list=UU3jmlL2rHbipq_2tDd5zDRA