My partner said he'd never let anything happen to his kid
But his idea of 'anything' bad is very different to yours.
I have some experience of this OP; before we had kids my then partner said he'd never hit as a method of punishment. So I thought we were on the same page.
Since then he regularly forcefully pulls my 3 year old away, digs his fingers into his back, pushes him over with his hip, grabs his cheeks hard and if my 3 yo is hanging off the couch he'll push him off hard to "give him a shock to teach him a lesson". But it's ok, because he's "not hitting him".
It all happens when he's pissed off, either because of work or because the 3 year old isn't listening first time (he's a toddler FFS!). It's vindictive, I can tell by the look on my ex's face that he's taking his anger out on the toddler, and I fear for our 1 year old twins. (I'm making plans to leave asap)
So just because your partner says the vague "will never let anything bad happen" doesn't mean you would be happy with what he exposes your child to.
Ignore what your partner says, and concentrate on what he does, which is leave his daughter upset to continue with an argument and not take a baby away from an abusive mother.