I just don't want to share a vulnerable space with men. Also, unisex toilets make it easier for sexual assault to take place.
The bathroom is more than a place to wee. On nights out when I was younger, I used to go there to escape creepy guys. You go there to cry. People talk about their boyfriends in there. It's a safe place to ask for menstrual products from other women.
I just don't want men in that space, when for drunk and vulnerable women it is actually a place you go for a bit of peace and safety sometimes. Also, women in some religions might find it super uncomfortable. And call me paranoid, but I just don't wanna be sat pants down with only me and a door with a shit lock between me and a man.
Finally, blokes toilets are generally just grimmer. Public toilets are nasty anyway, but men's are 100x worse.
It would bother me slightly less in a controlled work setting (like a toilet for a small number of people in an office), but even so, so many men pee on the seat and don't clean it up.