Hi.
Sitemaps are done per domain, that's for sure. And use of hreflang is just saying that "Here is a link to a different language version of the website you're on right now". So, it doesn't cover both websites.
Example:
You went to Paris and bought a map of Paris with labels all in french. As you're looking at that map, you have no clue what those labels say, since you can't speak/write/read French. Then you notice that at the bottom of this map there is a sentence "To buy a map of Paris in english, go to such-and-such store".
Now, does that sentence (reference) help you anyhow to translate what's on a map you're holding in your hands? No.
The same thing with sitemaps. Reference to a different language version of the website doesn't pass the content of that website. Especially, since you have all URLs in different languages.
Long answer short - you'll need to sitemaps.
Hope this makes sense and helps.