That depends on what percentage of your links are made up using those keywords. Negative SEO usually takes a LOT more effort than building a few crappy links to your website. The same has been done to some of my client sites in the past and it's not made any difference to their SERPs
Anyway, the simplest solution here is to login to Google Webmaster tools and use the Disavow Links tool so that at least they won't count towards your backlink profile.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=2648487
Hope this helps.