I guess I must have misunderstood some of the things you've said or put my own spin on them to mean that you were trying to be a presence in his life. I apologize.
I agree with Ledkr, though. You do need to avoid him as much as possible for your own good and recovery from your heartbreak.
Do you think you could arrange to take DS to his parents and they could drop him home so you wouldn't see the scumbag? Would you feel, in a bit more time, that his mother would be capable of caring for DD during brief visits? I guess I'm only thinking that because I trusted my DH's mum to care for our babies at 6 weeks old so DH and I could go out for a few hours to dinner or a movie. I was EBF and would leave a bottle of breast milk for her to use. If DH and I had split up, I still would have trusted her. She would never have endangered her grandsons.
Naturally, his mum would have to agree to never leave them unsupervised and if you can't trust her to do that, all bets are off. But wouldn't it be nice to know that you could just take the children, drop them off, then pick them up (or have his parents bring them) without having to see him or put up with his rudeness? He can always hide in the bedroom closet when you drop them off if he doesn't want to see you.
In a way, the power is in your hands. You make the decisions as to how the children get to him. Not him. He doesn't want your family or friends to hand off? Tough shit. If you say that's who will be doing it, then that's who will be doing it.