Hi
Have you checked for messages in Google Webmaster Tools? It could be a sign of a penalty I.e. Google devaluing a large portion of those 4k backlinks you have.
My advice would be to head on over to http://www.linkdetox.com and run a health report on your backlink profile.
If you need to clean things up a bit, start by emailing the domains that contain those "high risk" links and request removal then go ahead and disavow them.
After all of this start building back some authority by creating great content, sharing and earning solid backlinks from high authority sites.
I hope this helps, if you have anymore questions please ask.
Kirk Fletcher