Ok, don't want to be critical, but the way many of these posts are worded has me a bit 
It the way you ask dh to have the baby, or he offers to have the baby.
It all sounds as if the baby is your responsibility and dh very generously has baby for an hour or two 
The baby is his too. Does he say Can I have a few hours without baby on Saturday to you?
We have always worked it so that when both parents are at home, evenings or weekends, we are both responsible. Our kids are now older (6, 9 and 11) but we have always done this, turns to have a lie- in, whoever is downstairs does breakfast, whoever is home does lunch/games/homework etc.
We plan our own activites in joint consultation, so - I have a hair appointment on Sat, stick it on the calendar, dh doesn't book something at the same time, he needs to go into town for stuff, he gives me a heads up on Friday evening, I have kids while he is out. It isn't really about him going out with kids so I have space, he deals with needs of kids as they arise, and so do I. Because we are joint parenting, the kids don't migrate to me all the time. If I am busy, eg in the garden, I will say to kids, go and ask daddy.