If you issue the ultimatum, what are you hoping to achieve? He leaves her and moves in with you?
What if he says no thanks, it was nice while it lasted, bye?
What happens if he says he won’t leave her but he still wants to carry on with you?
What happens if he says not yet, now isn’t the right time?
How long are you prepared to wait?
I’m guessing your affair currently consists of the adrenaline and excitement of planning the elicit encounters and only snatching the best bits of each other, making the effort, possibly you being a sounding board for all his marital woes, without the mundane realities and practicalities of life and living together.
But I say yes, issue the ultimatum and take it from there.