They are bullying you and their behaviour is horrible.
Do not be afraid and try not to feel intimidated. You have done nothing wrong. You have not caused their behaviour.
You are under no obligation to be present when he has contact with his daughter.
Stay calm and don't let them get to you.
Make a list (perhaps get a small notebook for this purpose) of all the times he has had contact with your daughter, where the contact took place, and for how long. Note down if you were also present during the contact time, so you can demonstrate that he has refused to see her without you being there.
Also note, if you can, all the times he has not turned up for pre-arranged contact.
Continue to keep these notes.
Again, you are NOT obliged to be there to facilitate his contact time. You do NOT (and I would suggest should not) let him into your home, as he has been abusive towards you.
They are probably making empty threats towards you because you are not doing exactly what they want you to do. This is bullying and intimidation.
If they want to make something of it, then let them take you to court. I can assure you that if it comes to that, it will not go well for him.
Don't engage with them and don't get drawn into any arguments.
Sorry you are going through this.