You are trying to please too many people, it will never work.
One thing my DH brought it, after a crappy holiday, was that we sat down after a holiday and talked about what worked well and what didn't. As holidays are expensive, they don't happen often, so you want to get it right.
For us this helped us figure out over a few years what sort of hotel we like, what is a waste of money for us and so on. To the point we now don't need the conversation as we both know what is going to work.
From what you have said, the first thing that would come up is 'planning it is so stressful, nobody wants the same things'.
So how can you make it less stressful? Expecting you, your DH, older DCs and both sets of parents to like the same holiday is a very big ask.
You can just stop sharing the holiday and go just with your DH and/or DCs, not expect everyone to stay the same length of time, not expect everyone to spend all day together. Expecting an 18 yr old and a 20 yr old to enjoy the same holiday as their grandparents, or even want to go with their grandparents is very unusual and essentially setting yourself up to fail.
Attempting to plan the same thing every year will get what you always get - a load of rows and nobody happy. You have to do something different.