'He is earning the money while you enjoy the proceeds'
Sorry but that is BULLSHIT.
OP is a SAHM. Her financial contribution is in saving the family the (generally huge, especially for more than one child) cost of childcare.
And, as she says, doing just about everything else in the home.
OP has already commented that her earning power is a lot less than that of her H. So in fact she is probably making a greater net contribution to the family finances by SAHM-ing than working out of the home right now.
That's before you even get started on the longer-term financial and career sacrifices she's shouldering, like compromising her future earning potential in favour of his (leaving early because one of the kids is sick? - not Mr. OP!)
She's 'paying' the mortgage on their family asset just as much as he is.
OP, I do have some sympathy with your H. But it does rather sound as if he wants you home with the kids, as long as you don't have too much of what looks like fun doing it, because its not fair. Well both jobs are different, and for me personally I think the staying at home on the never ending childcare cycle is, mentally, much harder.
Perhaps you should reply by saying, ok, I won't go, but I want in return for you to promise me not to talk to your colleagues, have a laugh, go for coffee etc. no adult conversation in the daytime, because I can't have that and it's not fair. I wonder what he'd say.
I would ONLY SAHM if I were able to take advantage of the one single freedom it offers - not to have to be somewhere at 9am every day. So I don't think you're being unreasonable. And it's also for your children, and presumably why you bought the apartment in the first place.
Some compromise on timings should be reached, but fundamentally he needs reminding that there's a hell of a lot you're sacrificing to stay at home with the children, and no, it is not all made ok because you get to be at home all day. Quite the opposite in fact.