There is no need for any man to ever be in a woman's space...
Father with small daughter can take her into the gents (if she is too old she can go on her own into the ladies).
Bloke dressing up as a woman can go into the gents
Bloke who 'identifies' as a woman can go into the gents
there is zero need for a bloke to be in the ladies changing room / loo
the arguable exception is service staff (cleaners / plumbers / etc.) and if that is necessary it should be managed...
no decent bloke would ever force themselves on anyone else male or female, certainly not into a female space.
no decent bloke actually wants (or is really comfortable with) women in the gents / male changing rooms either (esp. in today's climate where accusations can be made).
It is really simple - you either have fully unisex rooms (like the loos in lots of garages - or the ones we all have at home! - one room / one door) or you have male / female options and based on your birth sex you go into the appropriate one...
no objections to any chap who wishes to put on lipstick and wear a skirt - plenty of cultures in which that is quite normal - but that does not make them female / a woman and they do not need to use anything other than the male loos...
the only reason a man could possibly wish to use the ladies facilities is if they are trying to make a point / force themselves on women / have a sexual perversion / etc. none of which are acceptable...
(and yes I am male - and all the males I know would also agree)