Was it $400 monthly or what?
If you've paid a 1 off fee of $400 for "SEO" then you did not receive what most of us on this site would call search engine optimization. In other words rather than being scammed, you just bought into something which was not SEO. You are right though in saying that someone willing to take your $400 for something he calls SEO is probably the kind of person who would ruin your site if you push for a refund (I've seen some dodgy companies do this to their clients in the past)
Just cut your losses, find a better SEO company and let them deal with whatever damage has been caused to your site by this other company.
Be prepared to spend quite a lot more than $400 though, if you're going to insist on cheap don't expect miracles.