"Look what you made us do. If you don't let men in the women's loos we will try to bully employers into an utterly impractical and inefficient change to having all single user rooms, and then everyone will be cross."
They won't succeed of course. There just aren't the resources or the space, and landlords won't be on board. In practice I suspect what will happen is if the accessible facilities are under-utilised, that will be used, and if not a store cupboard or similar will be adapted to make an additional single user facility.
Accessibility for disabled users does not mean all toilets need to be accessible, just a number proportional to the number who need it. Accessibility for people who feel unable to share with their own sex does not mean all loos need to be unisex, just a number proportional to the number who need it.