You cannot feel like a woman, because 'woman' isn't a feeling. It's a biological fact. It's like saying you feel like a brown eyed person, what do brown eyed people feel like?
No man can feel like a woman. He can only feel like what he imagines a woman to be. No one is in the wrong body, what do you think it is that's in the body? A soul ? That's a religious belief. You are your body. Many if not most adolescents feel 'wrong' in their body or wish it was otherwise, at some point don't they? That's normal. Plenty of older people too for that matter. Discomfort with your body or feeling it doesn't represent how you want to be is pretty normal. It doesn't mean you're in the wrong body. Plus, many of those who ID as trans do not have gender dysphoria. They have no problems with their body and no intention to change it.
I'm pleased you have the privilege not to care.about single sex spaces. For some women they are the only way to be able to participate in society. Respectfully, admission to being female and female spaces isn't yours to give away to men you've decided deserve it because they want it. And as you rightly note, there is no way to police which men feel as you describe, and which do not.
Finally, no one is trying to make anyone feel worse. Though some of the people who are distressed to find out that women are actual full humans with rights, had no interest at all in how we felt, when they took our stuff without asking. In fact, some of them gloated about it and told us to 'enjoy ur erasure'. So I'm not losing any sleep about them to be honest.