you poor thing.
I don't think there is ANY reason good enough for your dd to have to go with someone you believe will put your dd at risk.
You will have to stand your ground.
If and when he calls to arrange sunday (if he hasn't already done so), you can try these options.
The first being, that you don't trust him with dd after last time.
Which is the truth but will probably turn into a row, and if you relent, he may or may not tell you where he is taking dd.
Secondly,
You can blatantly lie, and say dd is not available on sunday (think of somewhere you have to be/go) and prepare for the angst. If he gets very angry, you can just say, you will have to see her another time. This will buy you a bit of time, to sort out how you are going to deal with this long term.
If he threatens court, then let him. If you truly want this to be done properly then he would have to go and get a court order. This buys you more time.
He cannot physically remove your dd from you.
If you think he's going to turn up, then be out. I had to do this once, as he was ignoring my dd's pleas.
In the end I went home, but got my dad to come with me.