When a woman goes into the men's she is taking risk herself.
When men enter the women's they are putting all women in a mixed sex space so they all bare the risk (even if he won't accept the TW are a risk then can't he see that letting some men in let's any men in).
Lots of males are 'at risk' in the men's boys, old men, disabled men, small men, ill men, drunk men. Does he think women should welcome them all into the ladies?
Toilets are the tip of the iceberg - does he think changing rooms, hospital wards, prison cells and hmmm boxing rings should become mixed sex too?
If men are unsafe around other men that is a 'men' problem not a women problem and men need to solve it.