OP, he may say he will not 'hurt' the younger ones, but none of you are emotionally safe with him.
especially when you are still suffering from his emotional abuse enough to be persuaded by a few posters that your 'judgement was clouded'.
Your judgement was not clouded; trust your instincts, believe your judgement and take seriously the reaction of your 11 year old DD. To downplay what she feels would be t effectively gaslight her now, and I am sure that is not your intent.
It isn't a shame that she doesn't want to see him; it is a triumph, and will be her liberation. Why should she see someone who winds her up and then threatens her in a cold menacing way?
Don't fall for his accusation that you are turning your DS against him: it is emotional blackmail designed to make you feel guilty and try and appease him. Or to prove him wrong by turning yourself inside out trying to reconcile your toddler with him.
Unfortunately this thread will now be filled by posters who feel they ca dive in late without having RTFT, and lacking the empathy to read your OP and think 'there must be something else going on here' and 'I wonder if the OP has now discussed the context'.
Blithe reports of the gruesome empty threats bandied about in healthy families are irrelevant to your situation.