Because they are men.
Male-free toilets aren't just about safety, though this is paramount, they are about dignity and privacy, too. You may be comfortable sharing a toilet with male-born people, but most women aren't. Women have had miscarriages in public toilets; they have had to clean themselves up after unexpected and embarrassing menstrual flow; women with urinary and/or faecal incontinence may have to clean up in them - this is humiliating enough even where there are other sympathetic women around; occasionally, if shops sell clothes but don't have changing rooms, they try on clothes in them (I used to do this in local Sainsbury - buy the items, try them in the hand washing area of the loos, then take them straight back if they weren't right - and I wasn't the only woman to do this); they go in there to cry, often to escape men who are being a nuisance - there are lots of reasons women don't want men of any kind in their sex-segregated areas.
For many women of faith - Orthodox Jewish, Muslim, Rastafarian and probably others - women aren't permitted to share these intimate spaces with men they aren't related to. Letting any man in means that these women can't use the spaces. This means that many of them will be unable to go out to any public place for any length of time. The "urinary leash" will be back with a vengeance for these women - do you think that is fair?
Self-contained cubicles aren't the answer - hidden cameras have been found in these, placed there by men, and also they need to be ceiling to floor. This can be dangerous in a medical emergency - if someone has a heart attack for a fit is it very difficult to access the cubicle to assist them. It can also be dangerous if a woman is forced into a cubicle and assaulted - no-one can see to raise the alarm or help.
Plus - how "inclusive" is it to TiM who don't "pass"? - men who genuinely have a dysphoria, but who, with the best will in the world will never truly look like women? How will it make them feel that some men can enter women's spaces but they can't? And how will anyone know who has had genital re-assignment without "genital inspections", which is a totally unacceptable thing, taking the dignity from everyone, whatever their natal sex or preferred gender?
The only way to reduce women's vulnerability is to keep men out of female spaces.
TiM either need 3rd spaces, or to address the very difficult problem of male violence. This is men's problem to solve - not women's.