I can see the difficulty.
And I think your DH has acted badly in the way he's handled this and you have to nip that in the bud now.. that cannot be allowed to happen again without consulting you.
But for me it would be a case of weighing up the pros and cons. Since you've said you will probably go but unwillingly. It would also be a case of carving out time for yourself to spend doing what you want.
Other things I'd be thinking about include.
Can you afford it in terms of annual leave and finances.
Is he desperate to go or just can't say no.
Is it that you feel one week already booked and now a second?
Might it be possible that he himself will find it all a bit wearing by the end of the week and be less likely to want to repeat it.
If you did go I would insist that you go out just the three of you on your own at least twice in the week.
Is the kind of place where you could take baby out with you on your own, go for walks, go on day trips?
Or is it that you don't enjoy the same kind of activities.
I feel that with a very small baby, you have quite a good excuse not to take part in those, especially in the evenings and could entertain yourself and baby quietly.
If its in a hotel, you'd have several opportunities to go off on your own.
Would they babysit so you and dh could have an evening off?
(my personal anxiety would be having to leave a small child in the care of a group of careless relatives insisting on having them around a pool or sea, especially if the people have proven to not be as careful and as watchful as a parent)
I don't expect you to answer all these particulars here, but just listing factors I would make my decision on. I mean, if it is really unbearable and youcan't stand it then say no. But if you've decided you can put up with it for DH sake, then go. But either way, make sure he understands he can't just sign you up for things liek this again.