Honestly it's not the case.
I've only the one but a wealth of experience due to being eldest of 3 kids, umpteen cousins and I've been caring for other people's babies since I was 14 inc being a nanny and childminder.
Ex was youngest of 4, no cousins, no experience with kids let alone babies and he went through a phase of feeling similarly.
So I encouraged him to hold dd more, do more with her and get the practice in - and magically
the more he did, the more time he spent with her and more relaxed he became, the more she "liked" him.
Do lots with baby, don't stress when baby cries - it's what they do and there's not always a reason, is there always a reason when you're in a bad mood? Of course not.
Yes you check its not hunger/wind/dirty nappy etc but sometimes they just wanna let off steam!
It's very early days, you're still getting to know each other.