Hi imjonny
I'm glad you have asked around to be honest. Like I said, I would.
You are right that even if you canonicalize ultimately Google will decide whether to rank a page it deems to be important and can ignore the canonicalization. If the canonical isn't bona fide then it could call in to doubt the other canonicals on your site which would be a strong negative signal for SEO and lead to a drop in trust.
So - it depends on what you want to rank for.
Let's say you have Product A on site A and you then have an equivalent checkout page on site B. Then you can't try and rank for the term Product A because it just isn't going to happen. We've already said that we will need to canonicalize that page anyway to the equivalent page on site A.
The only thing you can hope to do is rank for 'Delivery Options', 'Branded Delivery' The Big 'Delivery Option Site'
What you can't do is try and rank for the product names, It will be impossible. But then why would you want to? Surely the important thing is to maintain rank for site A's products with site B being more of a slave site - solely functional.
Ultimately you would be canonicalizing the product pages not the whole site so maybe there are other pages that you can add. Maintenance, Technology, How to etc But frankly they would suit Site A anyway because if I am buying a product I want as much info as possible before purchase, not on the delivery page.
Oh and don't create branded content for site B because once again you will crave up site A.
I know it's a big conundrum but I haven't seen anything like you are trying to do so can only generalise on best practice.
I hope that helps!
Regards
Nigel