Are there diminishing returns for optimizing a site? No. At least not as far as being penalized for organically achieving new rankings.
Now then, you say you're getting good high rankings for various search terms, but you're not seeing increased traffic. Let's figure out why that could be.
First of all, when you're checking you're rankings, make sure you're not receiving personalized results for having clicked on your own website too many times. Do things like make sure you are logged out of any Google accounts you may have, and add the code &pws=0 to the end of the URL for the searches you do.
If personalization is not affecting you, and you are achieving those ranks, then you likely have some other problems. There are 2 things most likely affecting you.
1st) Are you using the right keywords? If you're a company that offers a 'credit card fraud investigation service', you don't really want to be ranking for terms like 'credit card fraud'. That term isn't specifically related in its intent to what you are needing. Someone searching for credit card fraud is probably looking for information about, can you guess it, credit card fraud! They're probably not so much looking for people who investigate credit card fraud.
- Your search description snippets might not be working well for you. Make sure you have meta descriptions declared on every page that is ranking, or is of importance, which clearly describes the content on that page. If you don't have meta descriptions when someone see your site in a search result, they could be seeing a description that looks something like "We have been in service since...... keyword is what we do if..... we don't think that way". Basically they're seeing something that makes no sense in relation to what they searched for.
If you declare targetted meta descriptions, you're likely to have a better click through rate, and thus more traffic.