Removing backlinks, especially ones with money anchor text, will most definitely cause your rankings to decrease. I can only assume you did this because you viewed the links as spammy or something that could eventually cause you future ranking problems at an algorithmic or manual level. If that is the case, you probably made the right decision if you are hoping this site will rank for the long-term.
Instead of restoring those links, try to find a way to create a few new ones that you feel are safe for the long-term.