Oh believe me, I am not 'a cool wife' in the slightest! 
I just think there is a place for the child to spend time with her dad (breakfast is going to be what? An hour?) and then for them to spend time together as a family.
But then, my exh and the children did stuff as a family, but we also spent time with the children separately too. I go away for the weekend with my children separately too because they have different interests. As has their dad. We are still a good and strong co-parenting unit and we've talked about this and the positives of being able to spend quality 1:1 time with each of the children. You get to know them differently outside of the 'family unit'. You're not joined at the hip.
I don't see that as being 'cool', I see it as being normal.
Whatever you, Slender, the OP or I think or say, the dad does have a voice in this too. Maybe if there is some dysfunction in the relationship, then ensuring that everyone's needs are met, especially in light of such a difficult year, should be the priority. And that includes the husband's.
After all, whatever lies behind it, if this has been going on for 4 years, and has only recently come to a head when he has made it clear he'd like to spend some time with his daughter on his own, then he's hardly a total ogre.
Whatever the state of the OP's marriage (and that does need resolving), the dad and the child should be able to spend time together alone.