Same for me. Walked out of the cubicle in some gender neutral loos in a cafe, actually thinking I was reluctantly on board with using them in that instance, to see a man at the sink a few feet away - and the instinctive reaction was instant and absolute. Automatic risk assessment: he's between me and the door, there are empty stalls behind me, the toilets are at far end of a huge open room with no one in immediate hearing... I know it's a thousand to one, but if he had been that "one", I'd have been utterly helpless - the mixed sex space put me in a position in which I, quite simply, had to trust that he wasn't. And being forced to place my trust in a strange male, with all I know, when there are alternatives to this, is simply not good enough.
I've not actually gone back since - I'd have liked to, but why would I when other venues don't present the same slightly increased risk and stress? But why the hell should I have to self-exclude in this way?!
I just don't think men, including transwomen undergoing treatment, begin to "get" the impact of the male-female strength & size differential on how women navigate the world. OK, men may have play-fought, or real-fought, other far, far stronger men, as I have other, far stronger women (martial arts) - but being overpowered by the same sex is simply not the same as when someone is, quite literally, in a totally different (sex) class to you strength- and size-wise.
Most males, including most transwomen, are in a position of absolute, not relative, power over most females. With that comes, for almost all women, a deep-seated wariness that's totally natural - nature - and encouraged by nurture - how many bad encounters of what kind & to what degree degree have we each had (& it is a matter of how many etc., not whether we ever have...)
We deserve space from this, and from them.
When you recognise and accept this, the sense with which words like "inclusion" and "safety" are now often used is, frankly, insultingly facile and ironic.