Justice Swift addressed this in GLP v EHRC
39 .... would place too great a burden on employers, requiring them to either “police” the use of lavatories or risk prosecution for breach of the 1992 Workplace Regulations. This point is significantly overstated.
40 The notion that an employer or anyone else is required to “police” the use of a lavatory, person by person and day by day, reveals the application of a “logic” so strict that it is divorced from reality and from any sensible model of human behaviour.
A 'sensible model of human behaviour' is that people respect the law, respect sex-segregated spaces as designated, 'the good men stay out so the bad men stand out' and we all get along OK, as we did until a small group of unrepresentative, noisy, aggressive and sometimes violent men started demanding access to women's spaces as if it was their human right.