What an awful situation. PPs are quite right that your H is being coercive/abusive. It’s totally unacceptable. If you don’t want to have sex with him, you must say so. And he must respect that.
It reminds me of the Aesops fable of the North Wind and the Sun. If you coerce and pressure a person, they’ll naturally resist, hence your comment about not wanting to let him ‘win’ by enjoying it. Whereas if you make a person feel appreciated and desired, they’ll probably want to do it. He’s clearly not doing anything to make you feel special, if you feel like ‘a hole’.
But, it also goes both ways. It sounds like you’re not really into him, even if he weren’t coercive, due to your different personalities. Knowing you don’t want to be physical with him must make him feel undesired and miserable. In fact, it sounds like he may be depressed. And his lack of enthusiasm/grumpiness is escalating things further.
It’s a really tough one. First, do you want to stay with him? If so, you have young kids, which can really affect a couple’s sexual relationship. So, you should perhaps give yourselves grace and keep working at communicating, making each other feel special and having fun. Once the kids are a bit older, you may find more desire.
But, it sounds like you’ve never really enjoyed or looked forward to sex, either in previous relationships or for most of your current one. So, perhaps it’d help to ask yourself would you like to enjoy it or are you happy without it?
If the former, then perhaps a first step is figuring out what the barrier/solution is. Libido can be affected by lifestyle, rubbish lovers can be overcome by masturbation or maybe it’s more psychological (e.g fear of pregnancy or because you’ve been taught to view sex as dirty or unpleasant).
If the answer is you’re happier without it (and some people do prefer celibacy), then you need to be honest with your H. And you have to be prepared he doesn’t want to be celibate and you have to split.
Whatever happens, he must immediately stop expecting to have sex as some conjugal right (it’s not the 1950s) and start respecting your feelings. So, it’s probably best to make clear you don’t want to have sex for a certain amount of time while you contemplate how you want to move forward. Good luck.