Am disgusted on your behalf that your chair "forgot" to invite you to the first meeting where you would have been part of the decision-making of the times of the subsequent meetings.
Sounds like he did it on purpose.
Don't see how you COULD take along your daughter, seeing as how you'd have to leave again to take her to nursery - that would disrupt later things happening in the morning, wouldn't it?
I would stand your ground, send an email back saying you have one of two ways of attending the meeting: offering to either attend the early meeting bringing your daughter, or that you will arrive late to the meeting after you've dropped your daugher off. If he has another go, reiterate that your working hours are from 9am (check your contract about working outside these hours), and that it is just not an option to do it his way. Remind him about him forgetting to invite you to the first meeting and I'd also write "I wonder how essential my attendance IS at these meetings seeing as I was overlooked for an invitation to the first one."
And if your organisation bangs on about "family friendly" working practices anywwhere - quote those in the email too.
Breakfast meetings are a completely stupid idea. If the meeting was important enough to hold in the first place, then it's important enough to find time for them as part of the normal working day, surely? Not have to wedge them in somewhere as an afterthought.....
for you.