And you feel like it has spoilt your maternity leave a bit as you would've also liked to just be able to watch crap TV with baby, etc?
If you said to him either:
I think we should all go to the zoo together (or whatever) today or as pp have said:
'I'm off out so you'll have to amuse dd whilst I'm gone'
What would be his response... would he do it or would he make excuses?
I get that that isn't the point of your post as you want him not to exclude himself from plans at the weekend and that it's hurtful that he's asking you this question in a way that does excludes him, but I think in order to change things, or at least find out exactly where you and your dd stand, it would be good to challenge this statement and maybe not give him the choice to swerve it.
Clutching at straws, it could simply be miscommunication on both sides and because you do usually go out every day, he assumes that's what you like to do and that you're excluding/ don't need him, so he is really just asking innocently, ie, do you have plans (or are we doing something together?).... he's just not adding the last part of the question and assuming you know that's what he means.
Maybe he is phrasing it the way he is because he doesn't want to be rejected as he assumed your plans won't include him a you're out for large chunks of the day every week day?
I do really think you need to sit him down and simply tell him how that question makes you feel and that although you're happy to give him space to work during the week, on the weekend you want to do things together, even if that is just chilling at home.
Hope it works out. Either way though, you'll at least know for definite where you stand 🙂