"A person possessing the natural or acquired traits, such as strength, agility, and endurance, that are necessary for physical exercise or sports, especially those performed in competitive contexts. "
OK OP, I both run competitively (as a veteran athlete) and showjump competitively. Yes, I consider showjumpers to be athletes. You certainly need to develop your strength, agility and endurance to be a successful rider, its something you need to work on to improve. You also need balance and timing. You constantly work on these along with the skills of developing a partnership with a living animal.
Lets put it this way. I got on a downhill mountain bike on holiday for the first time in the French Alps last year, and within a couple of hours was happily racing down the mountain nearly as fast as most of the guys there. I picked up a table tennis bat for the first time on that holiday and was almost immediately able to beat my friends who had played quite a lot before. I don't get those crossover skills from running. Running is a good sport for the relatively uncoordinated. Its very straightforward, but over-dependent on muscle type, lactic threshold, and body composition, as opposed to development of difficult and varied skills.
I think your objection is using an animal as an aid to the sport. Many sports use "props", in fact most do apart from running and even for that, track athletics is still kind of artificial because it uses...a track.
In terms of age, in many sports, you need young muscles and joints to be at their peak. And the prevalent thinking is that people "should" retire when they are a certain age. So horse riding must fall into the category of sports which are dependent on technique, like archery, shooting and golf.
But god yes, you need a lot of strength for horse riding. Its really hard on your back, and your ankle and knee joints.