That said it sounds like there is a lot of acrimony and unresolved issues here. Try to sort it out rather than risk it drag out for years damaging your daughter or even worse...ending up in court which is the worst possible outcome for all concerned.
Try mediation - google NFM and book something. You both love your daughter and should see each other as natural allies - not enemies.
Don't use him as a babysitter for something important. Use a friend or family or paid babysitter.
Set specific contact times on regular basis, if he doesn't show then tough .
Keeping child up past a bedtime is not a massive welfare issue. But you need to formalise contact and not use him as babysitter for this kind of appointment when things are not good between you and you feel unable to be honest with him about why you asking him to be with dd.
I was going to link but not knowing where you are makes it difficult. Google lone parent helpline and find the one for your area.
The good news is no he can't just take her and if he is going to threaten to you don't have to let her out of your sight until he gets a court order saying you have to ( which will also say he has to give her back )
He has shown no consideration for her feelings, kicking off in front of her and keeping her out past her bedtime is out of line. Please please please keep a log of you interactions with him you will find it really useful.
If you are worried about this happening again, you need to have a legal agreement in place. Get some recommendations for a local solicitor in family law and speak to them about the issues you have. It may simply be something that you can negotiate with him/his solicitor without costing the earth.
A while back i had a doctors appointment and asked DD's dad if he could look after her while i was there, he i said ok. I told him i was going shopping as i didn't want him to know about my appointment so i said a friend and i went shopping to the nearest city to me, only about an hour away.
The appointment went on a bit longer than i anticipated so i called DD's farther to let him know i was on my way back and sorry i had taken so long. He then started accusing me of looking at properties for DD and i in the city. I said i was just shopping and as he didn't believe me he started swearing/shouting at me down he phone telling me I'm out of order moving an hour away from him etc. I said please stop swearing at me or i will put the phone down. He carried on so i put the phone down.
When i got home i called him to say i was coming to pick DD up and he said no i couldn't as i put the phone down on him. I had to go and plead/beg with him to be able to take DD home again. By the time he finally let me take her home it was abut 9 pm, DD was crying as she was so tired/hungry/distressed. DD was only about a year old at the time.
I'm so worried he will do it again if i go through the CSA. His name is on the birth certificate so he also has PR. Would i be able to call the police to get DD back or would they not help as his name is on the birth certificate? What sort of rights do i have if this happens again?
(DD lives with me and he see's her as and when he can be bothered which is normally once every two weeks despite living 10 mins away and working about 3/4 days a week)