I'm always amazed at the people who think looking after other people's children is remotely comparable to looking after their own. As far as I'm concerned, looking after other people's children is a chore, whereas looking after my own is a joy. How sad not to see any difference.
I worked full time when dd was a baby, but had no commute and did a split shift so that most of the hours I worked were when she was asleep - we had four hours of childcare in our own home in the morning, two of which coincided with dd's nap time. Then I'd go back to work in the evenings while DH stayed at home with dd. I also did all of the night wakings as I continued to breastfeed until dd was nearly 3. So yes, there were a couple of hours of active childcare that I missed out on, but actually I did do most of the stuff that a SAHP does. Same once she started school, as we didn't use any paid childcare from that point.
For me, the time I had at home with dd every day was definitely my time "off". My job was mentally and emotionally very demanding, so childcare responsibilities absolutely felt like a welcome break. Had I regarded that time as work, I suspect I'd have hit burnout after a few months.
Of course, I realise it would have been harder if dd had been a more difficult child or if I had had multiple children. For me, though, if looking after my own (NT) dc felt as hard as working in the kind of jobs that I'm used to, I'd take that as a sign that I wasn't suited to the SAHP role and start thinking about alternative arrangements. Of course, it's very different when you are caring for a disabled child or one with complex health needs, but I'd regard that role as more of a carer than a SAHP.
On the basis of my own experience, I don't see why the OP's DH couldn't go out in the evenings and do some part time hours after looking after baby in the day. However, there is no reason why he should do this, unless he and/or the OP actually want that. While I don't personally think being a SAHP is an actual job, it's a perfectly valid lifestyle choice if both partners are happy with the arrangement, and other people's opinions on the matter should not matter one jot.