Are social services pressuring you ? Are they a dinestuc violence specific team ? I found that the 'normal' family team sw who was involved after DD was assaulted didn't have a clue about the dynamics of DV and it was only when the DV specific team became involved that the children started to respond to them.
I'm with monet and would concentrate first on finding out what the issue is, and then taking it from there.
You don't really want to be in the position of denying access completely unless there is a very good reason for it, and it isn't just your daughter piking up on the situation.
If it helps any - my DS is 3. He tells me all the time 'I don't love Daddy' 'I don't want to see Daddy' 'I don't like Daddy' without any prompting from me. DS was 6 months old when we split so it's not like he ever knew us together, or has experienced anything to make him feel this way. Also - we are actually quite amicable and sometimes do things together with DS to try and make him feel comfortable.
When we are together DS always chooses me - wants me to do things for him, pick him up, hold his hand etc, and gets stroppy if we try to encourage otherwise.
It got to the point where he has having meltdowns when we were handing over so we made arrangements for him to pick DS up from nursery. The nursery staff tell me he is totally fine to go when this is the case, is happy to see his dad, excited to go sometimes even.
What I have put it down to is that given I do 90% of his care and he is used to being with me, that rather than hating his Dad and not wanting to see him, he just prefers being with his mommy, and doesn't quite know how to express that.
His Dad is with a new woman who has a child of similar age - I think he dislikes sharing toys, prefers the toys he has here, likes the things we do together.
Maybe it's a similar case with your DD. Maybe its even down to being nervous about seeing him now that things have changed. So it's not necessarily the worst case. and not necessarily a reason to stop contact altogether.
But you can't plan what you are going to do till yo understand what is actually going on.