It seems to me, that your husband is punishing you. He changes tact so as to cause maximum pain. He has a lot of hate in him and it’s causing his behaviour to be extremely ugly.
I don’t like saying it, but you aren’t the person he wants to be with. He’s never been realistic about trying to save this marriage. He doesn’t want to.
It’s likely the ow told him to ‘go back to your wife, I can’t leave my husband.’ But he’s wallowing in self pity. Instead of seeing the ow as a woman with few morals and a glimmer in her eye over their business associations...he is viewing her as self sacrificing. She has given him up (her true love) for both her family and yours. Hence, she is the good person - and you’re not even the slightest bit grateful for this huge sacrifice.
You never had a chance of getting your relationship back on track and I fear if you had, he would have made your life a misery.
It’s going to be very hard for a while. Make sure you surround yourself with love. Family, friends, pets. Keep yourself occupied. Seek professional support. See your gp. Realise that this isn’t about how good you are or something you should feel guilt over. You’ve done nothing wrong. You’ve been you. And that should have been good enough. It’s him with the issues.
The only way you can deal with him now, is to ignore. Anything he says is designed to cause pain, so try not to engage. Get your affairs in order.
You do not deserve this. Nobody does. Stay strong and know, many have got through this before you and many more will have to go through this in the future. You aren’t alone and you will be happy.