My brother was very much the same as the OP's husband when he was a new dad, works all day and then would rather do all the "house jobs" tidying up/washing up, take the dog for a walk etc not because he didn't want to spend tim with my nephew, but because he had never really been a "baby person" or had much to do with babies, despite the fact he and I have 14 younger cousins that we are all close to and spent a lot of time with growing up. I loved having a new baby almost every year for most of my childhood - I loved dressing them up and looking after them etc whereas my brother was only interested in the cousins that were old enough to run around in the garden with a football.
Once he had his own child, he struggled a lot at first because he found things too delicate or small so he felt like a big bumbling oaf, which knocked his confidence, especially when his partner or one of the grandmas would take over because he was doing it wrong. My brother is a 6ft2 rugby player - his newborn son seemed so tiny and fragile that every time he had to do something as simple as get him dressed, he found it hard because he was scared to hurt him by straightening his arms to get them into babygros if he was all curled up in his cot, his hands are big so he could barely fit a finger inside the arms of the clothes so even if he managed to get the baby's arm into the sleeve, unless it went on smoothly he struggled to get the clothes on properly and then even if he did get him into the babygro, he struggled to do up all the poppers because their small size was just too fiddly for him. He'd pick the baby up and if he couldn't position him "just right" then he'd panic that he'd drop the baby as he tried to readjust him in his arms because the baby was so small. He was terrified to bath him because he didn't want to hold him too tight in case he hurt him, but was always worried he wasn't holding him securely enough and was scared he'd lose his grip on a slippery wet baby and he'd fall underwater and drown etcetc
He got a bit more confident over time, and if he was left alone to do things with the baby, but it was a vicious cycle - because he was so nervous his partner was reluctant to leave him in sole charge of the baby, but the more she was around the more he let her do it because he felt like she was watching and judging him for doing everything "wrong", which meant that he got less practice and was less confident to do things alone and even if he did go out somewhere then there was always a family member wanting to coo over my nephew and hold him/feed him/change his nappy etc so my brother was never short of someone to do the jobs he wasn't confident with.
My nephew is now 18 months old and my brother is a much more involved dad because my nephew is bigger, walking, started talking a bit, and is a bit more independant so my brother isn't terrified that he'll break if he so much as looks at him too hard.
Also they can interact now - my brother is a huge football fan and even at 18 months old my nephew loves it - he's not quite got the attention for a full game, but he loves to sit with my brother in his own kit and watch the first bit and then he's more than happy to play with his toys while my brother watches the game and cheer randomly. You can sing songs and play games and play with toys together with my nephew now, even if it is just you building a tower of bricks and him knocking it down while laughing hysterically which my brother loves doing, watching my nephew bat at the toys hanging from his babygym playmat was boring to my brother so he struggled to muster up the enthusiasm everyone expected him to at the baby's progress in co-ordination, so he'd take himself off to do some busy work so he wasn't asked to pretend to be enthralled.