Love, that smug look will be wiped off her face pretty damned soon.. sadly. Pity her to be honest. I know you can't right now, but imagine the shit that it will be for her to realise in a few months/a year that actually all that she wished for might just kill her. Give her a totally pitying yet equally smug look back.
I totally understand what you are thinking about the children. I don't blame you at all. It would kill me to hand my son over to a bloke that has done nothing for him, has shouted frightened and terrorised him, has terrorised and destroyed me and sought to drain my entire life of anything worthwhile just so that I can be abjectly miserable and be under his control. Lucky, I don't have to share my boy with his feckless father. yet.
WRT your situation, AS LONG AS he is not harming them, then access will be fine. If the children are upset by him, or by access then take action.
I will say though that it's quite disturbing for children to be introduced to new partners too quickly, so perhaps you start with one day a week, then one day and an overnight and as long as the children are settled and happy you all work towards the shared 3 nights each.
If he is making advances towards you then make sure the hand overs are in a public place and ideally you have a member of family/friend with you if at all possible. Also tell him that you wouldn't touch him with a shitty stick having seen what he's stooped to, no matter HOW much he begs you. Say it in front of her if you must
All of what you describe is about control. Try to detach, don't ever get emotional about anything in front of him, that's what he wants. stick to agreements (if they are court appointed) if not then YOU tell him what the agreements will be until a formalised plan is in place.
YOU get to call the shots in your life now, not him.
Read the Lundy Bancroft book, and also Feel the Fear and Do It Anyway.