The gym is basically my safe haven [...] the panic attack that any conflict surrounding either challenging them or being questioning by staff would be enough to make me never go back.
And that is why, bananafritters, we must preserve single-sex spaces for women that blanket ban anyone born male from using them. It cannot be up to individual women having to speak up to challenge transgressors, society cannot expect those who are vulnerable and rely upon its protection to challenge those violating their boundaries.
The law has provided for a limited number of female-only set asides for our protection, privacy and dignity. Where these are provided as intended, very few men dare to transgress into them, because the social taboo is so strong.
It's so strong that not too long ago, abusive husbands would stop at the door to the ladies, because they knew if they went in they would not only be challenged by the other women in the ladies, but also by any men nearby. We policed each other on this and it worked. That's why we didn't need to have signs up saying access prohibited.
We already know that this social taboo is breaking down and those providing support to victims of male violence are finding their opportunities to help (which are often fleeting, rare and painfully inadequate) are disappearing.
So no, I wouldn't expect you to challenge a person like that. I would however consider writing a polite email to management asking for clarification, explaining that this was a chap who casually walked in to a changing room where women were undressing to chat to his friend without giving due regard to the other users.