Op I have just sat and read your thread from start to finish.
You need to take control back of your life.
Stop doing family things with him and the boys, do family things with you and the boys,
Sort child care out. So you don't have to rely on him,
Call CMS
He's basically having his cake and eating it, coming and going as he pleases. I can't remember if you said how old the boys are, are they old enough to walk to his car? Or to his house if you drop them off. Without the need for you coming face to face with him.
When he does have the children, do something for you. Something you can't do when the children are around.
Grey rock completely, only reply if it's regarding seeing the children, if it's something else. Do not respond. Not even one word.
For years and year I used to wake up to 80+ plus messages, long essay ones of abuse and how everything was my fault from my ex. He was an addict, though alcohol not gambling, and he destroyed our family because of it, it's draining,
I came across the grey rock method, and stuck to it strictly. I didn't let him step foot in my house (my space!) and I blocked him from all social media, my life very quickly got easier and better,
Now our conversations are literally "these are the nights I'm working, these are the days you've got the boys"
Now you need to start building YOUR life. Stop letting him past your front door. He left the family, he doesn't get to dictate the family life any more.