You'd think they'd teach this at some sort of "trans basics" class.
Pro tips for not having people say you're a man, if this upsets you:
Tip 1) Don't go up to random women and say "I'm a woman", or "am I a woman". This may generate a accurate response.
Of course, this doesn't apply to people like your driver, or that Edinburgh Rape Crisis manager, who actually aren't "upset" by this, but enjoy the confrontation, and trying to compel speech. (Recall that story of Wadwha at a party demanding this of guests).
It's win-win - either the woman complies, so you demonstrate you have power over her, or she does not, in which case you get to call her transphobic, demonstrating your power over her.
(Tips 2 onwards would be things like "don't apply for women's awards" or "don't try to enter female sports". It's amazing how you could lower the incidence rate of people having to point out that you're a man).