Part of the scoring in dressage is the score for "Harmony" between that particular horse and rider pairing. It's an understated term, but has a coefficient of at least 2 (which means whatever score the judge gives the combination is multiplied by 2 when calculating the final score.) A horse has to trust the rider to walk into the arena at all, let alone listen to the rider's aids over instinct -- this is why the particular partnership between a specific horse and their specific rider is important and why horses aren't interchangeable. Though CJ/CH could probably sit on any horse and produce amazing results, those results are best on the horses they work with, day after day.
Dressage, from Xenophon onwards, is about creating a movement from the smallest possible rider input. The quieter the "aid", the better. This is why the best riders look like they're sitting still. This is the heart of the sport. Charlotte Dujardin can lift a seatbone and move her outside leg back a fraction and there will be a beautiful canter depart, and that is what is glorious about her partnership with her horses (be it Valegro, or Pumpkin.)
To get to that point, the rider has to know the horse, to feel when they are even in a position to ask for something. It would be no use even asking for a horse to canter if you couldn't feel that their muscles were loose and they were moving with their hind legs properly -- you wouldn't get the result you wanted AND it would annoy the horse.
And to get to THAT point, there is a lot of grunt work. My Apple watch estimates my average heart rate during a ride to be between 160-180. You have to warm up properly both yourself and your horse and be flexible and receptive enough to work on those aids. Sometimes getting a steady warmup with all three gaits in their right place is a challenge! You have to be so receptive to your horse's state of mind and physical preparation, both on the ground and under saddle, to do this even remotely successfully.
As far as the idea this is the sport of the privileged, it's just not. Some people have privilege. We have an impressive array of PSG & GP riders at my yard. Some of them are wealthy. Others are working students, showing and competing the owners' horses for them while the owners come up the levels. That is how I started out, as a broke student shovelling poo & doing yard work for 10 hours per day in exchange for lessons. Many openings for working students exist, across all equestrian disciplines, and it provides learning in return for work.
And as far as thinking you're "too old" to start -- you're not! Go and give it a try!