Thanks for the answer. I seem to be only being penalized for these anchor text I am using in my searches. Some of my other searches I am still number 1 or 2-3. They seem to be targeting the keywords I used the most in anchor text. I only dropped about 10 spots in these searches .
I will try to go back through my link profile and fix the changes I made and hope this fixes the issue. Thanks for the answer.I hope it doesn't hurt me to change them again but I am going to go back to what I had them as with just my url or site name.
Live and learn.