Are your children young?
He says he wants his family back, it sounds to me as though he wants to go off and play and thinks you will be there to welcome him back with open arms.
The thing is op, he has always had his own way and this episode is just another (albeit a very painful one) demonstration that he has the upper hand. I have a feeling he may come back, these types can't stand the thought of you pouring all that love and affection on someone else.
It's a dreadful place for you to be, having to assert boundaries until it gets so bad you are pushed to the point of no return.
You havn't done anything wrong, he's a cunt, acknowledge that, no need to be polite anymore, he is treating you with contempt.
Until you act, there will be no possitive change for you, you can neither get over this and heal ,or be happy if he returns because you will hate him for it.
It's a no win situation.
He needs to know that actions have consequences and at the moment you don't feel strong enough to implant them. Try your best to look after yourself, sort out child maintenance and get in place fixed times he has the children officially, let him see that he is not a single man without responsibilities turning up when it suits him.
Lean on your family/friends if possible, it's time to get support and don't divulge anything to his family, they will side with him.