For us, it's the sheer busy-ness, crowdedness, etc., i.e. too many people, that gets us. Right from leaving home, roads are congested, the transfer bus from the car park to the terminal is full and standing, huge queues and baggage drop, huge queues in the security hall, queues for shops, drinks and food, crowded departure lounge areas, queues for boarding, queues for the toilets, when you get there, queues for immigration/passport control, congestion in the baggage reclaim, queues at the car hire desks. It's such a relief to actually get to your accommodation, as long as there isn't a queue at reception. Then queues and congestion at breakfast the next morning. Not enough sun loungers round the pool. Day out to the water park and there's a sodding queue for the plastic doughnuts (not enough of them), then you're queueing for every ride. It just goes on and on.
It's awful really. Everything is cramped and crowded, too many people, never enough staff, nothing is big enough for the number of people they're crowding in.
Never used to be so bad. We started going abroad in the 1980s and it was sheer bliss. Airports were far more pleasant places - never as many people, barely any queues, shopping, eating & drinking were a pleasure - not having to muscle and nudge your way through hoards of people. More leg room on planes meaning fewer people. Hotels had more staff, hardly ever queues, dining rooms of a sensible size in proportion to hotel size, plenty of sun beds, etc.
It's definitely a "people" problem, ie the number of passengers/holiday makers has increased massively, mostly due to population increase, but roads, airport facilities, hotels, etc., havn't grown in proportion and most have now adopted the "pile em in" mentality. It's just not pleasurable anymore. Too many people everywhere.