Strictly speaking, it’s not men (ie it’s not the fact they’re an adult not a child which supposedly gives them that “entitlement”). It’s adult males with a GRC. (GRCs aren’t available to children.)
Adult males without a GRC, which awards the status of “legal fiction female”, do not have an entitlement to access opposite sex spaces. (Other than under Stonewall law.)
The problem, as has been said on here, is that once an entitlement to access opposite sex spaces has been awarded to adult males with a GRC, there’s no practical mechanism for stopping adult males with the PC of GR but without a GRC from accessing that entitlement, nor of stopping adult males without the PC of GR from doing so either.
I’m sure you know all that already. I just wanted to point out the distinction between the PC of GR, and having a GRC. As having the protected characteristic of gender reassignment (applicable to under 18s) does not automatically confer the rights conferred by a GRC (restricted to over 18s), and the PC of GR is being deployed a lot in this thread to justify eg social transition.