Genuine question: why do you make him go if you know he doesn’t want to?
OP, I have similar with DH’s family. They go on 2-3 holidays a year together; they’re not the type of holidays I’d enjoy even if we went independently, but with my in-laws it’s like you and others have described - everyone has to be in the same room doing the same thing all the time. I don’t particularly get on with my in-laws at the best of times, and on these trips I feel absolutely surplus to requirements. I once managed to go 2 full days without any of them saying a word to me, when I decided that I wouldn’t initiate any conversation (but replied when they spoke to me). I also just don’t have enough annual leave for it all - they all work in schools so have absolutely loads of time off, but I don’t, and my annual leave is precious.
I was similar to you in that DH just “assumed” we would go every time (in other words, he wanted to go and didn’t give it a second thought). After a couple of years I told him I wasn’t up for it every time. We now go away with them once every other year for a week, and that’s more than enough (last time we went I took a stress-induced migraine and spent half the time in bed, which was actually bliss compared to socialising with them).
DH happily comes on my family holidays (which happen about once every 5 years, and my parents pay for us to go, whereas on his trips we’re expected to pay for his parents). The difference is that my family don’t sit in the same room the entire time; we use it as a chance for my parents to spend time with the grandkids while DH and I go for a coffee, or sometimes DH and BIL will watch football while DSis and I go out or whatever. Also we include everyone; his family don’t.
The best thing I did was saying no to going on every trip. If you think you can’t do it because of the fallout, remember that only toxic families have fallout when someone wants to do their own thing - healthy families encourage them to do it. In my case I was able to use annual leave as a reason, as well as saying that I didn’t enjoy it; actually after the first time that we didn’t go, DH realised it wasn’t the end of the world and is now happy enough.
My nieces and nephews have never been on holidays with just their parents; they’ve only ever been with extended family, and never had the experience of looking for fun things to do and then just doing them, because that’s not allowed. Don’t let your kids end up like that.