OP It sounds very likely that he has aspects of narcissistic personality disorder. If that is the case then the person you loved never ever really existed! It is this false person that you miss, but what you have to realise and come to understand is that it was always just an illusion.
When something like this happens it is very very like a death happening, you have to mourn the death of the person that you thought they were. It hurts like hell and you are confused precisely because his actions are making all of the betrayal and hurt much much worse.
Read up about covert narcissists and emotionally abusive people and look at the red flags. I would bet you anything that you will tick lots of the boxes.
While I understand that you don't want to let him go, you have to. He doesn't love you and probably never did as narcissists are only ever interested in their own well being. They can play the game and pretend and say the right things so that you think you are in a mutually loving relationship. In actual fact everything is only alright whilst he is gaining much more than he gives and it is what he wants.
His callous and hurtful behaviour has shown you the person he really is deep down at heart. Please have no illusions that he would have come back to you if the ow hadn't kicked him to the curb, he wouldn't. No matter how he dresses it up he was kicked out after a very short time probably because she realised what a lying tosser and a horrible human being he really is.
You have been conditioned by him to believe that he is your world and you can't survive without him, well that's simply not true! In actual fact it is the other way round he is the inadequate person who cannot survive without someone else making him feel important.
When he has said some of his most hurtful things has he had a smug look on his face? Have you looked at him and thought how can you say these things? He will have looked like that because he is actually enjoying your hurt and distress.
You should contact women's aid. What he is doing is tantamount to emotional abuse and possibly financial abuse since I bet you anything he spends a lot on himself but makes it so that you're not allowed to spend anything much on yourself. This is actually a criminal offence now which you can report to the police.
You are trying to keep this relationship together partially because it is all you have ever known and partially probably because you don't want your children's lives torn apart but that ship has sailed. All you will do if you stay in this relationship is allow your son to think that this is how women are supposed to be treated and your daughter that this is how her future partner can treat her and she has to accept it. I'm sure you don't want that for either of them.