So very difficult, it must feel as though you can't win, I bet you can't settle when they are with him and are too knackered to go out and 'enjoy' yourself when he does.
He sounds a useless fecker, not ploughing enough into creating a safe place for his kids to want to stay with him and blaming you at the same time for not allowing contact.
Fuck, he's a clown.
You will eventually see this man was nothing but a burden that brought zero to the table, a selfish shit who has abandoned his family in pursuit of his selfish desires.
Time will make this easier as your workload decreases as your children age, they will become your safe family and he will be excluded and no doubt in time he will become resentful about that too.
I hope you manage to get support from your own family, if you do not have that luxury try as hard as you can to pay for outside help, you need to conserve your energy as much as possible as you are effectively both mother and father now.
You deserve a medal as do all the other young mothers who are monumentally let down by these dead beat dads.
As for the pain in missing him, eventually that will wane once your confidence returns, most of all don't dance to his tune, you and the children come first now.
x