Yes OP, listening to and hearing that little voice - it’s a lifetime thing, unfortunately. I suppose when we lived in small communities we didn’t need to have this switched on so much. But I know any irritation, defensiveness, or sense of alarm is a little message to me. A message to me to re-consider, maybe to draw boundaries or retreat a little or a lot - usually saves me grief in the long run!
As for your particular issue, many moons ago, I also had a very short term relationship with a man who revealed his compulsion to dress in women’s clothes (sex was never mentioned though). He talked about it as a compulsion he found very distressing. This was before the so-called “trans movement”.
I genuinely felt for him, probably much like you. However, kind and naive, as I was, the idea of him turning up at my door, one day, dressed as a woman, filled with dread. I felt like a rabbit trapped in headlights.
Luckily, the very short relationship ended for other reasons, but I understand your dilemma is what I am saying. A very small minority of women might not mind, but if that is an issue for you – as it certainly was for me – you don’t have to go along with any of it. It’s that simple. As a previous poster said how many men would go along with this kind of scenario in reverse e.g. a woman who wanted to dress up as a man and have sex as a man with him? 🧐 .