I don't think there is any moral reason you should work, you have money independently, and you have worked already and know you don't like that type of work at least.
However, you do sound discontent with the decision, or rather, quite passive and hopeless in the face of it. Let's be honest, your husband is against you working, and now, with another small child (after the last one has just gone to school) the decision of what to do with your life is deferred- for now.
I don't say this to be mean, I have seen one of my very good friends struggle through exactly the same path recently, though, having never really had her career take off, not really like working after a few years, she was happy to hit on having children as her thing that would take her out of the workplace, and with a husband very adamant he liked her at home, everyone was happy- for quite a long time (she had several children). Now she is at home, bored, depressed, the children are all getting so independent and don't need her as much. She knows she wants to do something -but what? More time at the gym? Get the cleaner over more? Have more coffee mornings? All these things are fine, but don't really add up to a purpose, and I think this has left her open to an existential crisis tbh.
Now- if you said that you hated work, loved being at home, loved homemaking, it was your purpose- I would say go for it. But it seems something that has happened to you, almost by accident, and already your confidence that you could take a different path is eroding. That's a shame- remember you are only young, you do have a degree, there's so much you could offer in a few years time. Don't lose sight of that and start believing your husband's version of you.