Your homepage is probably more powerful than your category page and that is why it outranks it.
The category page, if it was strong enough to rank in the top ten, would likely be on the first page of the SERPs with your homepage - or instead of your homepage. That tells me that it lacks strength rather than relevance.
The first thing that I would do is make sure that I have taken advantage of all opportunities to build internal links to the category page. These would be links within paragraph text or a breadcrumb system of links throughout the site.
I often add lots of text to category pages. This can be details about the items shown on the page, relevant information at the top of the page, relevant sidebar information, or supplemental information at the bottom of the page. Use these to make links to relevant secondary pages within the category.
That's what I would do. If that isn't good enough then you need links from external sites to obtain enough power to rank higher.