Plenty of men don't want this. It's men that care enough to ask questions and raise objections or listen to women that are in short supply.
My personal experience:
I was at the Hammersmith Lyric a couple of years ago. The toilets had been made mixed sex. The door signs read something along the lines of "Toilets: Urinals and Cubicles" and "Toilets: Cubicles".
The first door was to the women's Toilets: Cubicles. I went in, no men, just a queue of women. A man came in after me. "This is nice," he announced, "I think it's a good idea. Nice and clean in here." While I was queuing, and when I was washing my hands after, all the women entering joined the queue. But some of the men just barged straight past it and tried cubicle handles.
It was hideously uncomfortable. Few of the women spoke, but many of us made unhappy eye-contact with each other. A couple of the younger women announced loudly how excellent and proper this new system was. But none of them spoke up when men barged past.
Afterwards I waited a long time for my friends in the outer lobby of the toilets area. The women's Toilets: Cubicles being the first door, everyone went straight for it. Some paused, puzzled by the door sign. Of the men who entered (almost all entered) about 2/3 came straight back out again to check the door sign. About half of those men shrugged their shoulders and went back into the women's Toilets: Cubicles. The rest went looking for the other toilets.
So about 1/3 of the men weren't bothered by being in the women's toilets, with women (if they even noticed).
About 1/3 of the men were confused by being in the women's toilets, with women, but chose to go ahead.
About 1/3 of the men considered it unacceptable to be in the women's toilets with women. That 1/3 was almost excessively older men.
I wish I had complained to the HL at the time, but I was new to this and very uncertain of myself.