fozzy 5 mins to have a quick visit to the toilet or shower or make a cuppa won't hurt, really. I mean obviously if you can't bring yourself to do it then don't, but i do think your DH is right.
You also could think about how things are going to be if you always give him 100% attention, I hate the "rod/back" comments but it is good to have a child who can amuse themselves a bit.
Having said that I have a friend who takes your approach - it is how she is and anything else would be uncomfortable with her. It does look like very hard work to me, but then i think she thinks that I am deprived of having a baby who turns to mummy all the time.
Which is another thing that occurs to me - do you let H look after him by himself without peering over his shoulder all the time? My friends children will never go to daddy if they trip over or anything, it's all on mummy, and that can leave daddy feeling a bit left out.
Just thinking out loud really.
For me it often feels like having a baby is like a weird game where the object is to not let them cry/stop them crying. But sometimes babies just cry because they're babies, not because anyone is doing anything wrong IYSWIM.
Sorry that's all a bit rambley!