Can men currently legally access single sex facilities?
Yes, in much the same way adults can legally access under-16 sports, and able-bodied people can legally access the Paralympics.
When category splits like that happen, they're not usually legally enforced - they're just rules.
There are vanishingly few actual laws enforcing that sort of thing, because they've not usually been necessary. Social convention suffices.
Hence the Equality Act only covers single-sex provision as necessary to make clear that it isn't barring it under the general anti-discrimination rules. The assumption was social convention would continue.
Those sorts of things tend to pop up as law only in international contexts - eg requiring separate facilities for female prisoners of war.