Yes, that's the problem.
It seems to me (IANAL) that Lady Haldane got it wrong but I don't know if that judgement can be contested.
The practical issue is that despite the law, providers of services which should have has the exception applied so they remained single sex didn't - they just rolled over to 'gender' (often as not self id'd) despite there being no real basis for 'gender' as an exception to the usual rule of equality.