You'll probably get some different opinions on this. Your site link architecture and the design of your archive/search/category needs to be assessed to figure out the best solution.
If I had to pick one or the other for the nofollow attribution, I'd pick the Archives over Categories because they have less keyword value. Generally, I'd say no need to use it. Sure, you might say that those pages don't have much value since all they contain is a repetition of content on your other pages. And some may try to use the "nofollow" tag as a way to prevent any link juice from flowing to those pages. But PageRank sculpting using this tactic isn't really effective.
It would be more effective for your SEO to redesign your blog/website sidebar/footer links so that you have "less" links pointing to your Archives and/or categories.