I have broken a contact order but with much older children and after years of literally having children ripped out of my arms screaming because they didn't want to go.
I wrote to the father (I can't speak to him because he is so abusive) and told him that the oldest child, then 10, was simply refusing to go, I told him I was aware that the order was being broken and if he wanted to take it back to court I would ensure that she had her own solicitor appointed by me that would fight on her behalf. I took advice from a family friend, who just happens to be a judge who handles these cases in another area, before I wrote.
I did tell him that I thought it was a 'phase' and that his handling of it may make the difference between having a future relationship with his daughter or not - he made the right choice and, after several months, she began seeing him again but not aver time, she chooses when she wants to go.
The youngest girl decided she didn't want to see her father at 8, he had verbally abused my husband as he dropped her off, it later turned out that he was drinking again and the abuse gave her the excuse she had been waiting for to stop going to stay.
His contact order was different to the other child's (different dad) and was only for 1 hours weekly but had gone onto staying contact over the years on the advice of SS, as he proved himself to be ok. This meant it wasn't quite so bad, the contact order made 5 years ago was for supervised contact, so I could arrange to meet him in the park or something for an hour, if he had been difficult over it. She is nearly 10 now and has barely seen him over the last 18 months or so, she doesn't care and if he turns up she will go out for a walk with him but won't go to his house. He is supposed to ring her but mostly doesn't bother for weeks on end, I think he has his own issues going on and she is no longer a priority.
With a child of 4 I think, as heart breaking as it is for you, I don't think there is a lot you can do without evidence of some sort of abuse. I do think that bad feeling from adults can cause more upset for the children and you should ensure that you never speak ill of his dad in front of him.
I have 3 dad's to deal with, 4 children (I am the grandma with residence) it can be very tricky if the dad's are difficult, and I don't have the relationship issues that parents have in place in these situations.
Please think long and hard before you make a decision you may regret dearly.