Let's rephrase that...
'AIBU to think I shouldn't have to inconvenience myself in a very minor way to reduce the risk of an accident that could cause myself and others life changing or life ending injury'...
You do not know what the rider knows.
The horse may be typically bombproof, but have just coped with a chasing barking off lead dog, a rapidly approaching concrete mixer, drum spinning making an awful racket, shouty rumbly bin lorry, spooky child on a swing appearing and disappearing over a hedge... (all real examples btw) and have his nerves absolutely shredded.
The rider can feel those tense bunched muscles, the thudding heart, the flicking unsure ears, and know that the tiniest thing is going to spell disaster and they'd, not unreasonably, like to get home alive.
Then here comes you, your run is by far the most important thing in the universe so you don't slow down, why should you. You don't shout out, you're out of puff, you're chuffing along slapping feet and dressed in dayglo like some sort of animated hazard warning and sounding like a cross between a clapping seal and a dragon.
Horse sees you before the rider has chance, in his panic he leaps, spins, drops the rider on the tarmac and takes off, smashing through the windscreen of a family car of four, killing self, driver and passenger and leaving two kids orphans.
You're fine though. You aced your PB. Well done you. Off you pop.
If you don't want to offer courtesy and consideration to others, run round a track.