I really do feel most concerned for your children in the middle of all this.
You have won the battle now. It is time to make sure that your children are freed from this conflict; to relieve them of the terrible responsibility of having to side with one parent or the other. It is a dreadful feeling for children, especially when feelings are running so high.
Make your decision - do you want to stay with this man or not? Then get stuck in to the necessary steps to achieve whatever you decide. But if you do decide to stay, please do not put these poor children into a similar situation again. And this is very likely to happen as this is a difficult man.
I am not criticising you in any way; but wanted to flag up that the risks to your children are not just physical ones of falling off a staircase, but long-term psychological and emotional consequences.
I lived in a home where my parents were waging perpetual psychological warfare, and, whilst we were physically well cared-for and there was no violence, the long term results of having to be on one side or the other and feeling helpless and manipulated have stayed with me. It is agony. Please make sure your children do not finish up having to endure this. You are an adult with choices to make - they have no choice - they just have to drift along in the wake.
As an adult now I realise that in fact they were very co-dependant - that the "game" they were playing was their lifeblood - but we children were the victims.
As you can see I have good reasons to be concerned about this - and maybe you may feel I am over-stating things - but I wanted to throw this into the mix. I hope you will treat it as positive advice.