Sorry to hear this - natural to feel a bit nervy around the pony even once the underlying cause of the behaviour has resolved.
Things I would say - prevention is better than cure and if you can avoid putting yourself into a vulnerable position in the first place around horses that is much better - so for instance our trot up zone for the vet is in the exposed corner of the car park which is very windy - very common to see horses get quite literally 'flighty' while being run up and down out there (plus of course the reason they're being trotted up is usually a suspicion that something's hurting them so all the more reason for them to be wound up). I always ask one of the yard grooms or the yard manager to trot up if I can - they're just much more experienced, strong and calm than me so no shame in getting someone more equipped for the task to do it. Plus I always put a bridle on for better control, in fact with my less reliable pony I'd take her absolutely everywhere in her bridle or a control halter, to/from the field, the arena, hand grazing her, whatever I needed to do in hand. Just not worth the risk of her getting away from me IMO. Equally another one of mine used to get really quite territorial in her stable, I would never ever groom, muck out, tack up in there, always tied her up on the yard to work around her, even where that was more inconvenient e.g. in the rain. I'd also never walk in with food in my hands as that seemed to blow her mind, her hay would go in while she was tied up outside and food in a bucket would go over the door. People on the yard used to take the piss and ask why was I so cautious when no-one had ever seen that behaviour from her, but the whole reason they never saw it was I careful to avoid putting her and me in that situation in the first place!
I'm not saying this to criticize you or blame you for the incidents, just more to say that trust your instincts and if there's an easy precaution to take that makes you feel safer then do so even if it doesn't seem 100% objectively necessary. It's easy to get annoyed with ourselves over anxiety and just try and shake it off but with horses there is a certain logic to the fear, they are big dangerous animals after all so I don't think trying to reason yourself out of it is always the best way, if you're anything like me you just end up arguing yourself into being even more scared by imagining all the worst case scenarios!
What can also help is getting some professional help and groundwork lessons, again people on the yard would laugh at me for needing a lesson on how to turn my horse out, such a routine/basic activity but actually that mare was a right PITA to lead in hand (hence bridle everywhere) from having been badly started as a youngster and had never learnt proper in-hand manners, having a few sessions with a really good pro taught me what I needed to know and gave me the techniques and confidence to handle the situation safely. Some pros that focus on groundwork are more natural horsemanship based, some are traditional, I don't think it really matters so long as it's someone you click with and draw confidence from, now the pony is no longer ridden it might be nice to do more groundwork based things with her and feel happy and relaxed doing so?