It sounds to me like he thinks its HIS holiday, so he can do whatever he wants, spend time with his family, catch up with friends, and you should just suck it up OP.
Why have you spent the evenings alone in your room with baby all week? Is there any reason why, if you're staying with his family they won't babysit for you? Or have you just assumed that it should be him who asks, as they're his family. If that's the case, then don't wait for him to ask, ask them yourself, and then tell him that MIL or SIL or whoever, has agreed to babysit while we go out together tonight.
I do think that you need to make it clear to him that you want to spend time with him in the evenings, and that this is a FAMILY holiday, where he gets the chance to catch up with his family, but that he doesn't get to leave you alone to take care of your JOINT children, while he goes out with his mates. When he went out, why did you shut yourself away with baby, instead of going and spending time with his family?
As for his mates not messaging their wives, he's forgetting that while HE is on holiday, his mates live there, and so their wives are probably used to being left alone for the lads nights out, and probably invite their girlfriends round for a girls night. Whereas he just cleared off out, leaving you on your own with HIS family.
I do think it's likely that you've sat and allowed things to fester a little, rather than using your words to tell him what YOU want out of the holiday though, so make the most of the last couple of days, and get out together as one BIG family, which is surely why you went there in the first place.