YANBU. You need a break and he needs to spend some one on one time with his daughter.
From when DD1 3 was around weeks old, DH would get up when she woke on a Saturday morning, bring her in to me for a feed then I'd go back to sleep and they wouldn't reappear until she needed her next feed. I used the time to sleep, and as time went on and I was getting more sleep during the rest of the week, I'd have a bath and catch up on some reading or TV. When we had DD2, DH would deal with both of them.
Saturday mornings in our house are 'Daddy Time', even now and our daughters are 9 and 11. Over the years they have spent thousands of hours in parks, cafes, charity shops and various 'coffee & cake mornings' in Church halls and the like. Initially, DH hadn't been 'in charge' of DD1 on his own before, so it gave him a gentle push to spend time with her on his own. It really boosted his confidence with them and helped him to form really strong Daddy-daughter bonds. Plus I felt appreciated, got the extra bit of sleep I craved and got a bit of time to myself, which makes DH's life better, cos as he says; "Happy wife: happy life!"
If you can brand one weekend day morning "Daddy and DD Time" and get him to buy into it (It'll be great bonding time for you! She'll really appreciate it as she gets older! She'll look forward to spending this special time with you! You'll enjoy it! etc.) then you're winning! Good luck OP xx