Honestly? Given the background if I were you I'd be fighting as hard as possible NOT to allow contact.
Have you been forced to provide contact? What legal advice and support are you getting about this?
If there was abuse in your relationship (and like others I feel this is likely) you need to get that recognised so that you can access legal aid. You may want to consider contacting women's aid, ncdv, barnardos etc who can provide support where there's issues like this.
It's abusive behaviour towards your daughter. Get your dr on board, get everything in writing and EVERY TIME he ignores her needs get it recorded.
Sorry but I think you've been wrong to force contact with him. He is bringing nothing positive to your dds life, she is plenty old enough to be listened to if she doesn't want to see him.
Like hell would I have let him back in my child's life after fucking off like that regardless of the circumstances of her conception.
What he tells people, what they think is the LEAST important thing here. Your dds health and happiness have to be your priority.
If he shows up uninvited at your home you tell him to get lost! He has NO right to do this and if he persists you make a complaint of harassment to the police.
What exactly happened when he came back into her life? WHY did he after so long? (I suspect a new girlfriend who rightly thought he was a shit dad and he tried to change her mind) has he been paying maintenance? Has he sent birthday and Christmas cards/presents?
Frankly no dad is better than a crap one especially when crap = abusive.
I'm shocked you let him just take her after several years of no contact! How well did you know him even when you got pregnant? A few weeks apparently? That's crazy just handing her over!
I think you need serious help regarding what are healthy safe boundaries for your dd.
You barely know this guy. I would not be letting an 11 year old who doesn't even have the confidence to say no to an activity that could cause her permanent harm be alone with this guy.