I think some people here live in another universe 
Ok, this is not the best way to start a relationship. Yes, he may simply be having his cake and eating it. It may be the cliche everyone has decided it is.
But wait... I live in a small village. I can count at least 4 couples who I don't know personally but know of who have got together as OW/OM (sometimes the woman was married, man was single, sometimes other way round, sometimes both were married.)
Outside of my village I know of other couples who got together where one left a partner and family, and of couples in my parents' social circle who did the same; all starting off as OM/OW in some combination. Many had been having the affair for years, decades, and waited for their children to be grown up.
I am not saying this does not cause hurt. But to tell the OP that it's doomed, that he will repeat the same behaviour blah blah, is not entirely accurate.
It might be, it might not be.
OP you yourself have said that if you give him an ultimatum he will not come back to you. If that is what you believe, that's your answer.
You will only know by carrying it out.