PR and medical matters

[Pumpkinpie11]

I'm looking for some advice on what I have to inform my ex about regarding medical matters for my DD.
The background is that my ex was given contact which was to be supervised by me in court. He however decided he wasn't happy with this after several no shows and said he'll only see her if he can take her out alone. Due to this he hasn't seen her for several months.
He has recently started declaring that I have to consult him over all medical appointments and that he wants to attend.
Do I really have to take a stranger to the dr every time she's ill?
I should say that my DD is 2 and has a physical disability which requires constant treatment. We rarely have what I class as 'medical' appointments but have 2-3 therapy appointments a week which are held in my home and one is paid for privately. He is telling me he has a right to attend these appointments even when they are in my home.
I'm aware I should inform him if medical matters but to what degree? The last (and only) time I tried to 'inform' him he decided to swing his weight around and tell me he can stop her having an operation. He backed off when I told him he'd look a prat in court against a neurosurgeon.
Can he really interfere with the well being of a child he doesn't see?