I think the major point is that when you are concerned about men coming into women's spaces, taking women's jobs, invading women's sports, it is very difficult to spare the time to worry that men may be uncomfortable that a woman is in one of their spaces?
My view is this. Everyone needs to grow the fuck up and behave in toilets. That means don't piss on the seats, remember to flush, wash your hands, don't throw rubbish on the floor. Use the facilities that correspond with your birth sex, and don't make people who don't conform to that sex's stereotypes of how they dress into something that ends in a fight. Men's toilets should be safe for all men. (biological ones) and Women's toilets should be safe for all women (biological ones)
There is a tiny difference though, and that concerns passing. Lots of TW think they pass as women. They reinforce each other in this (as do the handmaidens) and so when a 6' chap in a dress and lippy is told by a woman with courage to get out of the ladies, they are shocked - because they think they pass (I am being charitable here. There are other types of men that use women's facilities for other reasons). There are some trans men i guess who may, IRL, actually pass as a bloke, or an older teenage boy. Buck Angel i guess and some of the others that we see in ads and so on, muscles, tattoos, etc. Often the pictures are posed in a way that we don't see the hip to waist ratio, the shoulders, hands and feet, so it is often difficult to tell. My guess is that a) they don't want to use the men's for nefarious purposes (maybe they do?) and b) they don't make men uncomfortable.
so while i am happy for them to use the ladies, i can well understand that some other women may not because of how they look (to that i say: sorry, it's difficult but i believe that is correct), and i can understand that using the men's may be a better option. I'm not really sure, it isn't discussed much from the trans man or man's pov.
Trans men aren't taking opportunities away from men in terms of sports, jobs advertised for men (maybe rape crisis centres for men need to check on this?) etc. Basically they are not loud and shouty and in your face and so determined to have all the attention on them all the time. Elliott Paige doesn't seem to be experiencing any sort of euphoria in public, i hope that in private it is different.
I liked the "running away from / running towards" analogy upthread. I need to ponder on that.