You just have to get your head around the idea that a holiday-with-children has absolutely nothing in common with the concept of a holiday as you know it 
As a family, I think we do holidays really well. I’ve found self-catering much, much easier than all-inclusive because they can eat food they will eat, at their own meal times. Being able to stick (roughly) to a routine of meals and naps keeps things on an even keel.
I prefer weather at the milder end of the spectrum, though dh would prefer it equatorial. Over heated, dehydrated cranky kids are not fun.
We’ve never used a kids club either. We’ve just slowed down to a pace that works for the dc. In cities, we might visit one attraction, bookended with playgrounds and parks. Or instead of queuing and climbing the Eiffel Tower we picnicked in view of it, chased pigeons and soaked up the atmosphere. It’s lovely to get long stretches to just chat and be with dh, so sitting on a bench sipping coffee while the dc explore the puddles is no hardship.
I think dh and I are very considerate of each other. We will slip off in the morning with the dc so the other can have a lie in; if one of us wants to see an attraction that would be boring for the dc, the other will mind the dc. We fight over whose turn it is to wash up, in a good way.
As the dc get older they can tolerate our historical/cultural interests for longer. And this year they picked a few day trips that we genuinely enjoyed too. There’s always a little bit of longing on our part for more grown up time, but then we’re always aware that they probably won’t want to come on holidays with us for that much longer.
I’m a bit more prone to getting overwhelmed than dh, probably because so much time with the dc is a novelty for him and not more of the daily grind. I love the holidays but I have to admit that I love the routine and structure that school brings too.