Solost, I know you haven't acknowledged me on this thread, but just to pick up a point someone made and it is what I was getting at last night, now is the time to get a solicitor involved, now, right now.
He is messing with you both, you are both his toys and with you two hating each other, he gets of scott free and it's all a game and he gets his ego boosted.
You are now technically the OW, he is with her, he has made that choice, it's time to accept it.
It's not acceptable for him to keep phoning you and texting you, why should he? It's harrasment you are nothing to do with him anymore. I am all for absent fathers seeing their kids, but this cannot be at his beck and call. He lost all rights to make demands on their and your times when it suits him, when he shacked up with OW.
Part of the reality of running off with OW, is you leave your wife and kids behind, no more strolling in, putting them to bed, bathing them whenever he feels the urge and is missing them, tough he gave that up.
If you are ever to move on and get stability in your childrens lives and yours, they need set access really, especially with how things stand now, it will go on forever otherwise.
If he wants to come back he will, if he won't, he won't you are currently not helping the cause. Time to get firm, see a solicitor and begin sorting out formal access and money arrangements. Then just stay away from them both, ask him not to contact you out of access arrangements.
He is a dickhead and right now he has you where he want's you.