There is no legal requirement to provide single-sex swimming but it is lawful because it's a proportionate way of meeting this legitimate aim (it doesn't have to be a 'need' as such). As soon as you let some men in it fails to meet the aim and therefore cannot rely on the single sex exception.
It's like arguing that it's OK to let some men into a female only swim session at the local leisure centre because there are separate changing rooms. It removes the justification for the single sex sessions in the first place.
But if the excluded men are suffering no detriment, @PencilsInSpace ,and if they are facing the exact same exclusions as women (ie no entry to opposite sex pond unless trans-ID'd), then the providers would have no need to rely on the single-sex exemption because there would have been no discrimination or detriment on the basis of sex.
All they'd need to establish is that there were three broadly equally resourced ponds, and that men and women faced exactly the same access rules: Entry only permitted to two out of three ponds unless you have a trans ID.
I mean, I really want the women's pond to be women only, but I can see the argument that the trans-inclusive arrangement is legal. If that is what the users genuinely want (and perhaps there are grounds for legal review of the adequacy of the consultation and decision process, IDK), then in justice I can't see why the law should prevent it (so long as there are changing facilities that permit safety and privacy, and a baseline level of security arrangements to prevent sexual assault).
Some questions are correctly a matter for people, rather than the law, to address. If there is no consensus in society on the trans issue, we should expect a multiplicity of trans-inclusive and trans-exclusive provisions and associations. The Equality Act sets the parameters in which these can occur. It doesn't delegitimise all trans-inclusive provision by fiat.