Google also likes pages that load quickly. I think you need to balance the "view all" canonical tag with the effects on page weight and load time. I would stick with what's working - lots of paginated sites canonicalize to page 1, even though Google prefers the view all version.
