He's handled it very badly, and as such I'd rather it was him that was asking for advice and considering how to fix this, not you.
However, sometimes being right isn't as important as being happy - provided that happiness is genuine and not actually you being made to change your view to someone else's 'happy'.
My post is going to assume that he is a good man who doesn't have any feelings of entitlement over sex. For most people sex is part of the deal when you embark on a relationship. Very few people will be content in a sexless relationship, and as such it is unfair and unrealistic to expect the absence of sex not to cause problems. However, once a week is fairly good going with three young children including a baby. I think most reasonable men would feel quite lucky TBH. If your H shows any signs of feeling that he is entitled to sex regardless of how you feel; if he doesn't do his fair share around the house or with the DC, the following advice really doesn't apply.
Being continually rejected is hurtful. If you really are making your DH feel like a pervert, I can understand why he might have lost his control and ended up having a go. It was the wrong thing to do but if it's been going on a while and you're both suffering from sleep deprivation, I can see how it might have happened. It may be the case that what you're seeing as angry demands is, from his perspective, hurt and fear that you no longer desire him and no longer care about him.
At the same time, when you have three children, including a baby, it can feel that everyone wants a piece of your body and your time, so that there is nothing left for you. The expectation of sex on top of this is just too much and can feel like a chore. A partner who keeps bringing up the lack of sex can make you feel very stressed and sometimes worthless if the implication is that you are less of a woman/wife.
I'd pick a quiet time when you are both calm to talk about it - over a meal with a nice bottle of wine is a good time. You can use the food as an excuse to take a few seconds to consider what you want to say if you need time to think about it or don't want to blurt out the first thing that comes to mind.
Start off by saying that you understand his frustrations and you really don't want him to feel rejected or hurt. You love him and you want to find a solution that works for your relationship. Then tell him how you feel and why. Explain that you need to feel like a desirable woman before you can feel like having sex. That means having sleep and a break from the domestic treadmill of child-related and housework related tasks. You need time to relax and rediscover yourself as separate from DC's mum. He can help that process by making sure he pulls his weight around the house, making sure he looks after the kids so that you can go for a walk/visit a friend/do a hobby, etc. Feeling sexy comes from within. If you feel good about yourself and your life, you feel more attractive. Your task as a couple is to get you to that place.
FWIW, I think the consensus is that the average couple has sex twice a week, but it's really a meaningless statistic as it says nothing about age, length of relationship, presence and ages of children, job demands, etc.