Our sex segregated spaces and services are protected by the EA: Schedule 3 Part 7 para 28, explanatory notes here.
Also it is legal to stipulate that a single sex job is not open to TIMs if it's a genuine occupational requirement and a proportionate means of achieving a legitimate aim: Schedule 9, Part 1, General, explanatory notes here.
These exceptions are buried deep down towards the end of a very big piece of legislation. As you'll have noticed, law in its raw form is incredibly difficult for ordinary people to decipher, so it gets interpreted into more human-friendly 'guidance' (and we all know who's been writing the guidance, don't we 
) and then all the different organisations (public bodies, businesses, charities and NFP, community groups etc.) turn the guidance into 'policy'.
Most orgs don't have the time, money or inclination to pick over the actual law and decide for themselves whether the guidelines they are using interpret the law correctly. For most, this is just box ticking they have to do before they can get on with their real work. So they'll pick something off-the-shelf that seems to be widely endorsed and they'll base their policies on what eveybody else is using and trust they are complying with the law.
No org is going to go off-piste on this unless their whole raison d'etre clashes badly with the trans agenda. It's too risky, most especially for the masses of small charities and NFPs who are reliant on government funding to survive.
And trans is incredibly complex in all sorts of ways. It's taken me years to get any sort of grip on what it is, what it means, how it's legislated and the effects it has in everyday life. And still, I keep coming across new angles I hadn't considered. If you just want to e.g. run a gym, make some money and not get sued, you're not going to touch this with a barge pole, not least because you haven't a scooby what any of it means. You'll gratefully grab whatever guidance you're spoonfed by the 'experts' and build your policies in line with that.