Some families get on really well and are happy to go on holiday together in this way. Some can put up with it but would prefer not to, and some absolutely hate the idea. It’s much easier to tolerate in-laws that you find hard work for a day or weekend, than it is for a holiday, and if you’re working, sometimes the thought of using a substantial part of your annual leave with in-laws is miserable.
I went away once with my MIL and wouldn’t do it again. I wouldn’t go away with my SIL and family (she’s alright but her husband and children are difficult company). The thought of wasting precious holiday (time and money) in their company when I could be having a holiday I actually enjoy rather than grit my teeth through, is also miserable.
I think it is perfectly reasonable of your husband to choose not to go on holiday with your family.
Whether it’s OK for you and the children to go without him, I’m more ambivalent about. It is quite a big deal. I think he probably feels that you’ve chosen your family over him. This may not be entirely rational or reasonable but it does seem this may be a watershed moment in your marriage.
He may feel that you always prioritise your family over the marriage. Or he may be controlling and resent you having a support network/life outside your relationship with him. Or it may be something in between the two.
You should talk about this. Not in a confrontational way. Ask him why he’s so upset/angry about it and try and listen to what he says without getting defensive. Then take some time to think about whether he has a point or whether he’s just being a dick.