OP, I'm so sorry you're feeling down about this. Having a baby can have a huge effect on a relationship and many couples struggle.
As you can see from the thread, there's no single answer about how important sex is, or how frequent it should be. It's a complex and emotive subject and a bunch of us on the internet really can't get a very clear picture of your marriage.
It was shitty of your H to say you'd let yourself go, and that must have hurt, but we don't know if that was a one-off, or if it's part of a larger pattern of being shitty to you.
OTOH, it's really shitty to be pressured by your partner for more sex that you feel capable of performing. It's possible your husband is experiencing some medical issue, or a mental health issue, that is affecting his interest in sex. Being pressured can then make you feel less and less interest in sex with your partner. And just because that pressure is coming from someone you love, and it's because they crave emotional intimacy with you ... well, that doesn't necessarily make it feel better.
I'd suggest that couples counseling could help, maybe combined with individual counseling for yourself. This is more likely to address the complexities of your situation effectively than a bunch of us on the internet who are only seeing a tiny sliver of the picture.
If you decide you need to be in a relationship with more sex and that's non-negotiable for you--that's fine, you are absolutely allowed to feel that way. But just keep in mind that in any long-term relationship, people's sex drives can go up and down tremendously due to age, health, life situation (e.g. bereavement, new baby, work stress) ... it might be worth considering whether you could develop other ways to feel intimate with your partner and secure in yourself in the long term, or the same pattern could repeat all over again with new partners. (I don't mean you have to give up all hope of having an active sex life, but perhaps consider whether you can balance it out with other things.)
Best wishes with whatever you decide.