If the event isn’t a test of human endurance, ability etc, then it shouldn’t be in the Olympics. It’s only take seriously as a ‘sport’ because Princess Anne used to ponce about on a horse collecting adulation as an elite ‘sportswoman’. Even when I was a kid I could see it was a load of bollocks.
Anyone who has ridden horses for any length of time knows that it can take a LOT of physical effort. Not just for the however long it is they are competing but every day, often hours as they may be schooling many horses, then before getting to elite level mucking out, grooming, tack cleaning etc.
It has been calculated that depending on the type of riding you are doing, your own weight etc, it can burn 250 - 550 calories an hour just doing the riding, with dressage calculated at around 450 calories. In comparison, doing gymnastics burns somewhere between 250 and 300 calories per hour
Skill levels and your own body control are crucial; letting your head tilt just a couple of degrees to one side inadvertently can completely ruin a movement, letting your leg shift back a cm gives an entirely different signal to the horse, even tightening one buttock a tiny bit more or less than intended can affect the horses movement. To get this degree of control over your own body you need to train - hard - for years. I know it looks like the rider is just sitting there & the horse is doing all the work but I can promise you it's very hard to do the work while not showing it.
Some history on why dressage is considered a sport. The basic control over a horse to get it to do different things like changing pace, stride, turning on the haunches (pirouette) and side stepping on request were all part of cavalry training. Schooling in what is now called dressage goes back to Tudor times, and prowess in "Menage" as it was called then was greatly feted during the reign of the Stuarts. Horses were used in warfare until well into the 20th century, so the ability to ride and control a horse to a high level were valued abilities - Modern Pentathlon is based on the 5 skills considered a requirement for all Army officers (Horse riding, shooting, fencing, swimming, running).
TLDR - dressage is a sport which requires high levels of fitness and skill, takes many years of very hard work to master, and has been highly respected for centuries longer than Princess Anne has been alive