OP, if you're still reading, I still do think you should go and talk to your younger DC's counsellor about them going.
For a few reasons.
Regarding yourself, I'd be damned if I'd let an abuser drive a wedge between my child and me. No matter how awkward the day would be, but if otherwise my relationship with my child was good, I'd grin and bear it.
As for the younger DCs, it's definitely more complex, but, and this is where you should look for professional advice, it could either be triggering or it could be an opportunity to be in a context where abuse is not likely to happen and to act out and demonstrate how we can be strong around abusers.
Although we don't have to live with them, sometimes we do have to deal with them and face our fears. This is for the counsellor, the children and you to judge.
I'd certainly be ready to leave at the slightest sign of distress or abuse, anyway.
Finally, I'd definitely talk with my DD, as openly as possible, about all the issues surrounding this man and my reservations about him going and the effect the youngest are still suffering.
Sometimes it's easy to underestimate things we didn't go through.
So, openess is key here. She won't know unless you talk to her, and suddenly not going will feel like a shock and a betrayal if you haven't let her have the opportunity to understand your point of view in this matter.