You may well still be slightly under his control. He uses your DD to control you in whatever way he can, using her as a pawn to get to you. And he always will do, as long as he is able. Abusers don't often change, and the one thing they crave most over all else is control.
You say your DD loves exdp to bits... but you also say she doesn't want to go to his, returns from his upset because she is ignored there, does not really value any gifts he's given her, considers you and your DP her parents, and even says that she doesn't feel safe in his home?
What is your poor DD getting out of her relationship with him, apart from two very bad role models? From where I'm sitting, it sounds like sweet fuck all, at best.
My guess is that she says she loves him because she is only 4, and a sweet little girl, and he is one of only a handful of people whom she really knows. But do you want THAT man - the man who strangled you and hit you, verbally and emotionally abused you, and neglects his own DD - to be one of the few adults who impresses upon her little mind what a man is supposed to be??
You KNOW that he is an abuser. He MAY not have abused her - yet - but he certainly neglects her, confuses her, and messes her about. And I would argue that showing such minute and occasional interest, failing to turn up when she expects to see him, and everything else you've described, is emotionally abusive, as well as extremely confusing, already. And you posted that she says she doesn't feel SAFE sleeping over at his! If he hasn't begun to act just as cruelly toward her as he done toward you yet, well... I know abusers, and my best guess is that he will do soon enough.
You CANNOT continue to allow your daughter to see this man, if there is anything you can do about it! Not only is it extremely unduly stressful for you to be forced to deal with an abusive ex and his bullshit (when you should have cut 100% of contact years ago), it is also NOT GOOD FOR HER at all. At BEST, he and his girlfriend are teaching her terrible life lessons - about what adults/parents/men/women should be and how they should behave; about what a loving and/or healthy relationship is; about what love is and how people behave toward those they claim to love; about trust and trustworthiness; about how a man should speak to and treat his wife/girlfriend; and more I'm sure - but at worst, they are neglecting her when she is with them and possibly emotionally or verbally abusing her in some way as well already!
I'm sure your DD says she "loves" her "daddy" or whatever, but from every other detail you've shared here, she does not, really - or at least she is very confused about what love is and how it feels. Or maybe she loves him but doesn't particularly like him? She certainly doesn't seem to love him in a healthy way that we understand. If you were to cut all contact, as you should in my opinion, I'm sure DD would get over it quickly enough. Perhaps refocus her interest in exp onto your new DP and strengthen her relationship with a decent man who does not abuse her mother, a good man who teaches her the right lessons about what men and relationships should be. And even if she struggles with missing exdp for quite a while, that struggle is, again in my opinion, far preferable to the struggle she will feel when his neglect inevitably turns to proper emotional, mental, and verbal abuse, manipulation, and even possible physical violence and then REALLY fucks with her head!
As a complete layman and total non-expert who just happens to have personal experience with abusers much like your exdp, I cannot urge you strongly enough to cut all contact with this man, from yourself and especially your DD! And fuck what other people think about being "one of those mums"! They can mind their business, for one thing. This man strangled and hit you, verbally abused you, destroyed your property, intimidated you, and on! You need to do everything you possibly can do to make sure that he is never able to so much as try to do anything like that to your daughter! Personally, I'd rather be "one of those mums" than one of those whose shitty ex partners abuse their DC!!