Oh god, no.
We've taken MIL on holiday with us several times and each time it's been hard, hard work. She faffs, she whittles, she refuses 'forrin' food, she tells me what the DC should be doing ("they can't possibly need more sunscreen, they had some on this morning, you're so silly with the sunscreen, when I was young we used turkey oil to baste and we're all fine"). It's tiring. I come home from holidays with her needing valium. Plus, she's richer than us by a million miles but expects us to pay for everything, even to the point where she'll order things in flight on the duty free and say "oh, he'll take care of that" pointing at DH. And DH is such a soft shit he'll smile and take out his card and not say "piss off, lady, you're richer than God".
We've taken my Dad and his DP to Scotland a couple of times, but they're quite different and will bugger off and do their own thing, read the DCs a bedtime story and take them for a tramp outside but are generally far better at entertaining themselves. I think it's because they're both vey active (they're both dancers so gad about all over the UK and are very independent still).
We've never been away with my Mum and her DH because the one time they stayed at our house over Christmas we could clearly hear them having 'relations' the following morning and, frankly, that was enough to put me off sleepovers with them for life. They take the DC away from time to time and it works out far better.
I see nothing to be gained from our families spending holidays together.I love my parents, I love MIL. But holidays should be relaxing and fun, not trying to please others and, selfishly, I don't want to spend my holidays making MIL happy and acquiescing to her demands. I do that the rest of the year.