This man has manipulated you, he knew you fancied him, was it an affair ?
It sounds so.
The fact is, he's not matured enough himself to realise what he had, a 38 year old child, who convinced himself he deserved to demonise his wife because he caught feelings for an attention seeking 23 year old, sorry but at your age, love and the pull of finding a partner to reproduce is strong.
Now he should have understood that a young woman of 23 deserves the full attention of her partner to have children with, not be having to worry about 3 kids and an ex wife who you're not sure he still loves.
He did that, he took away your chance for the most special time in your life to be reduced to worrying about if he will sleep with her again at the the next change over with the kids.
You don't believe this but he is manipulative, and coersive and extremely selfish, you had agency but are very young. This pregnancy seemed unplanned and he sounds irresponsible, that could backfire in more ways than one.
I hope you have a good support system around you, I think you will need it.
He has made you think he is the victim, he really isn't, he is not a very kind, fair or loyal person, remember that.