I don't 'fear' trans people, or men in general. I'm a former athlete (I still train, but don't compete) in a sport that had precious few opportunities for women, so I've spent most of my life training, and sharing changing rooms with men. I don't care who I - personally - share changing rooms, bathrooms etc with. I'm in the quite unique position of being physically stronger than a lot of men and extremely capable of looking after myself, which I'm sure influences that stance.
I refuse to judge all trans people - or any other group - by their worst, or assume that they have nefarious intentions. I don't think 'trans women are women', or that it's possible to change sex, although I'm happy to call someone whatever makes them happy, if they ask me to.
With that said, I'm a firm believer in single-sex spaces being restricted to a single sex. No trans women, no men, nobody that isn't a biological woman in women's only spaces.
In purely practical terms, anything else is impossible to police. It's not possible to absolutely define what makes someone trans, so it's unfair to put employees in the position of making that determination, and undignified for those being judged. Service providers should be legally compelled enforce their single sex spaces at a policy level, and the government should be explicitly clear that there are no circumstances in which someone can claim to be anything but their biological sex in order to benefit from something related to how they 'identify'.
If the trans population explodes, third/fourth spaces will become necessary. But I don't think we're there yet. Trans people will have to live with the fact that the world can't be expected to make disproportionate adjustments for such a tiny minority.
I expect very few of us will have encountered a biological male in a woman's space. Fewer still will have encountered one there for purely nefarious purposes, and even fewer will have been/will be attacked or assaulted in a changing room/toilet. But that doesn't mean we shouldn't take every precaution possible to prevent these things from happening. Most of us won't have a serious car crash, but we always wear a seatbelt, every car has an airbag etc.
I don't care who I change with, or who sees me naked, but many people do and it's manifestly unfair to deny them their dignity based on a choice made by someone else. People should be safe, but they should also feel safe. No matter how progressive society gets on this issue, there will always be a significant number of women who won't feel safe or comfortable sharing spaces with men. Trans people, or people who don't care can be given the choice to use a third space if that's where they're comfortable, but we must never take that choice away from women, or men.
It's pretty rare on MN, but it's certainly possible to oppose various trans issues without wanting to segregate society or live in a constant state of fear/oppression.