OP you've a lot of good replies here. Note how even the posters who haven't RTFT and excuse his behaviour then come back and change what they say when they've RTFT.
You are being believed here and I'm adding to the chorus. I'm also 100% behind what winterdewinter posted. It's what I came to post.
In terms of moving forward, you need to implement a "no secrets, only surprises" rule with your DDs as of today. And keep repeating it. A surprise is a birthday present. A secret is when someone tells you not to tell mummy something. I told my kids that "It's our little secret" and "Don't tell mummy" and "We'll keep this just for us" were code words that meant "TELL MUMMY!" I would not say this in front of DH because he'll counter it the minute you leave the room. Your daughter may mention it to him anyway and he'll counter it. Never mind, keep saying it.
Also play the NSPCCs Pants song (it's on YouTube). The Pants song is fun and catchy! Sing it and dance around and clap (it's hard not to)!
Even if he's not a physically sexual predator these two things can help in other situations too. I also told mine that anybody wanting to touch their private parts (as named in the song) was a code to tell me too, or a teacher and me. Note, "code" is not "secret code"!
Those are things you can do immediately.
I would also start forcing time when you're all together instead of 1-on-1 time. If he gets upset by it because he's overwhelmed, tell him to take time off by himself to relax if he wants. You need to be inserting yourself all the time. It likely won't be comfortable. But reducing this 1-on-1 dynamic asap is essential. Have stock phrases ready that you want to be together, it's nice to all be together as a family etc. And don't back down.
At the moment I would not discuss any of this with him. He is selfish and manipulative (even if he has good sides too) and no doubt you'll be imagining it, he's only helping out, you don't want him to have a relationship with DD, you'll be accused of being unreasonable in multiple flavours. Simply start doing things differently, with a smile. If you suspect he'll get violent then don't do that, get out.
It's something you can start asap too but perhaps needs you to get your head around it all a bit more (re read winterdewinter's post and then read it again!). It can't come a day too soon though.
Ultimately, OP, this situation isn't healthy for any of you and you cannot allow yourself or the DC to be exposed to it. He's controlling, but there is control you need to take here.
Never forget what that midwife did, or how she did it. She went straight past him. No nonsense, no discussions, no pandering to his feelings, no worrying he might feel upset, no worrying she was being unkind. She focussed on what was best for your baby and just did it.
I'm sorry you're in this situation. None of the above is me implying any of this is your fault - it's most definitely not. His behaviour is not in the range of normal and that's not of your doing. He's systematically disempowering you as a mother. I want you to know there are ways of reclaiming some of that power to counter him. I am 100%. behind you, like I think everybody else on this thread is too.