OP your instincts are right: leave this relationship. Maybe you need to figure out how, but don't let that put you off.
Read up about gaslighting and DARVO. As a start I think you'll recognise a lot.
You're inadvertently enabling him, because you're in planet "Let's treat everybody with respect and behave decently" and he's on planet "This works for me". When they come together, only one is satisfied. And only one is on the receiving end of the sort of subtle abuse that makes them doubt themselves.
The way out is to literally get out of the relationship (do not attempt joint therapy). Until then, remember that whatever he says or does the underlying aim is to suit himself, distance yourself emotionally (takes energy) and go about your life in away that works for you. One that makes you feel good.
So if you think him saying those things in earshot of DSC is damaging, unfair, not true, go in and tell them that you don't know why he said that, it's really not true. You're upset he would say that, but not upset they're there. For example.
You could turn off your phone and go about your day with the DSC, tell then they're going to have to tag along a bit, but you'll see if you can make it not too boring. Take control back. Don't be waiting for him to deign to inform you if anything. If they're still with you when he comes home from work, his problem. If the mother contacts them then she can meet you wherever you're at to collect them. Do not be bending to accommodate anybody other than your DC and DSC well-being. If the adults are inconvenienced then their problem.
These are suggestions of examples, but the bottom line remains the same: he will not change and if you stay in the relationship you'll become increasingly unhappy because he's abusive. You're allowed to get on with things and not wait for him to give you the green flag to drop his kids off. He has you literally right where he wants you: waiting for him. Fuck that shit OP. Get your life back.
It will feel strange to prioritise yourself, but you're not being unfair in these circumstances. With someone else you prioritise them and they do the same for you and it works. With people like DH, they just take your kindness and trample you into the ground, slowly, over time.