I can't find a link to this video right now, but I know last year Matt Cutts posted a Webmaster Central video responding to questions about how Google treated adult websites in its algorithms. Matt stated that they don't treat adult websites any differently at all in their algorithms.
He did admit that he thought doing good white-hat SEO for them is much harder than other verticals however because many websites and properties will not be thrilled to link to adult websites, and so linkbuilding and getting exposure can be harder if you don't have a good network, but that there was nothing algorithmically against you.
That sounds pretty reasonable to me - I can't think of why they would arbitrarily penalize such a large category on the web, other than it's hard to do whitehat work for them. It sounds like you have the right tactics down already though.