If it wasn't work, any childcare worker wouldn't get paid for it.
My DH was a bit like this too, until I had an unexpected CS and he had to take time off to do what I always do - the school run with the older one, be at home, be organised (decide what to eat, think ahead, get stuff out of the freezer, get more than just 4 pints of milk when he shopped, etc). You all know it, the list goes on. He didn't do any cleaning/changing the sheets/paying the window cleaner/gardening/phoning the doctors, etc. So, actually, not nearly as much as I do.
After 3 weeks he suddenly exploded and shouted 'I'm SICK of doing everything around here!!!' That's when the conversations about 'I don't know what you do all day' etc stopped. Though we did have a chat about how under pressure he felt as the sole wage earner.. (he's self employed). And I do get that one, which is why I don't criticise his work or how he does it.
I do see that what we SAHMs do may have flexible hours and conditions sometimes, and time with our children (irreplaceable), but childcare's expensive, and I couldn't earn enough to even pay for the childcare, PT or FT. With tax credits, we're financially better off as well if one of us is at home FT. (As well as a million other reasons.)
However, SAHMs don't get set time off, sick pay/leave of any kind or holidays. It is relentless.. but then I'm saying stuff most of us already know. But unless you've been there, you couldn't possibly know!!
He may have been joking, OP (bad one). You can either take him to task over it and offer to swap him, or just try and forget it after a few hours fuming.