Yes, Justice Swift was very clear in several places in his judgment that separate facilities for men and women, where required, mean exactly that: separate spaces for men and women, and he took his lead from the SC ruling that sex means biological sex.
If I'm reading him right, I think he was emphasising the importance of the intention behind laws and regulations - the SC ruling was clear that the intention behind EA2010 was to identify 'men' and 'women' by biological sex, and Swift does the same with Workplace regs - it would be irrational [my word!] to suggest that separate facilities for women may also allow some men:
35 The obvious albeit unspoken premise of regulation 20 is the provision of private space for each sex for reasons of conventional decency.
37 It is clear from this that the objective of regulation 20 is that men and women should use conveniences in separate rooms, not together in the same room.
38 [] provided to meet the obligation under regulation 20 should be trans-inclusive. Rather, that employer has adopted a policy or practice to allow some biological males to use the female lavatory. An employer would not comply with the obligation under regulation 20 (to make sufficient provision in separate rooms containing lavatories provided for men and women, respectively) if he permitted the room for women to be used by some men and vice versa. That would go against the purpose of the regulation.
[my emphasis]
Apart from some occasional weird stuff🙄, I think his ruling is very matter-of-fact about the interpretation of the terms 'man' and 'woman.' wherever they occur, not just in EA2010.
I have no idea what weight his judgment has in the Grand Scheme of Things, though...