You would not set the canonical URL on the product page to your preferred category, that would send the signal that instead of having the product URL rank in search, you would prefer one of the product's categories, which would be incorrect.
It sounds like since you've already set a canonical URL for certain categories, so you're on the right track. For instance if you have 5 categories with very similar lists of products on those pages, it sounds like you are canonicalizing 4 of those pages to the 1 most authoritative of the 5.
In summary, you would only set a canonical for a product page if you had the same exact product spread across 5 different product pages. For example, if you had /mens-glasses/versace-primo/ and then for some reason every color had its own page because of your system, such as /mens-glasses/versace-primo/black, /mens-glasses/versace-primo/blue, /mens-glasses/versace-primo/red - then you would need the canonical on the product page back to the main page without the color subfolder.
To address a part of your question, it's generally ok to assign the same product to multiple categories if that's useful to the user. Unless you have a massive site and you're maxing out your crawl budget, I don't think you need to sweat too much on duplicate content issues, unless you're receiving really strong negative signals. For instance, Moz might show that some of your pages are duplicate, but it's more informational, and you don't always need to take sweeping action.
One strategy to fix duplicate content issues on category pages, is to write unique content for each category page and make the category page an actual destination page rather than just a navigational page.