Thinking about it from her perspective, maybe she feels something like this...
"Mum and dad have split up. I spend most of my time with mum. I feel safe and comfortable with mum. She knows me inside out, all my likes and dislikes, what to say when I get upset, how to sing my bedtime songs. I don't know dad as well as I know mum. I love him but he is not my mum. Sometimes when I'm with him and he does something wrong, like singing my bedtime songs wrong, it makes me really want my mum who would do it right. When I get upset he doesn't say the right things or reassure me convincingly. He doesn't make me feel safe and comforted, instead he always phones my mum and she comes over and picks me up and sorts everything out so it is all fine. I now don't really trust my dad to look after me because he can't deal with my upset himself- he always phones mum- and I am too young to comfort myself. It makes me really not want to see him now because he can't comfort and reassure me when I get upset, and even my mum thinks this because she comes to get me. I don't trust him to look after me like mum does because he doesn't trust himself and my mum doesn't trust him."
I think a lot of the problem your ex's confidence in himself being able to reassure her and calm her down when she is upset. Like what some other posters have said- their children cry for them but their h's don't entertain the idea that mummy is available when it is special daddy time. They both need the confidence to trust your ex.
I think that you need to have a good chat with DD and her dad, together, about why it is difficult for her to stay with him. Let her voice her opinions, make her feel heard but gently insist that it is very important that she spends time with her dad. Ask her how she would like to do this. With you their too at first? You could spend time just the 3 of you (DS at your mums) but with you in the background and dad doing the 'caring' but you are there for reassurance if needed but always encouraging her to go to her dad. This will encourage their relationship and help ex see his daughter when she is not sad and angry so they will be able to have fun together in a relaxed way.
Then a day out on their own? Doing something really fun together? You could suggest things like giving her an old t-shirt that smells of you/with your perfume sprayed on to take with her. Gradually building up the trust between both of them in a way that feels ok for everyone.
I would also tell her that during these arranged times with her dad that she can phone you if she really needs to (but try and get dad to use this as a last resort), but that you will not be coming to get her until the arranged time is up. She needs to trust that you will come and get her but it is unsustainable that this is at the drop of a hat. So you have a prearranged time eg 1 hr, 2 hrs, 5pm etc that she knows about and has agreed to beforehand, and that is when you collect her.
This is how I would handle it, might not be right for you and will probably be very difficult to start with and you might not make much progress for a while.
But it is so important for her and your ex that they have a good relationship, and while I totally agree with posters who have said that you need to pay attention to her feelings, I think doing it this way means that you can do both- relationship is maintained and improved in a gradual way that she feels comfortable with.