Yes, fair point - my initial wording may have been slightly too strong to allow for the rare exceptions where flexibility actually benefits the sex class of women for whom the spaces are designed.
On the letter of the law you could certainly have a go at arguing the inclusion of young male children with their mothers in single sex spaces constitutes indirect discrimination against women.
However, assuming that one of the justified reasons for having the single sex space in the first place (let's say a swimming pool changing room to borrow from the code of practice) was to increase the uptake of women using the service, though, you would be likely to lose and this would instead constitute a tolerated and lines departure from the baseline rule (the exception that proves it, if you like). This is on the basis that allowing young male children in actually contributes to that benefit for the sex class of women for whom the service is in place because allowing them to bring their children with them will likely increase female uptake if the service. It would also be unlikely to affect women's safety, women's sense of privacy and dignity etc. though all those things could be legally argued. This is why I said "likely" to be direct and indirect discrimination in my original post.
Those things would take you no further at all in arguing that males with a characteristic of gender reassignment should be included because it in no way furthers the original aim and reason for having the single sex space in the first place. That is, permitting men with the protected characteristic of gender reassignment in the space for women had no benefit to the women, is likely to decrease rather than increase their uptake, pose a threat to their safety (not only from those men who genuinely consider themselves women but because it in effect allows all men in, defeating the purpose; conversely including every single young male child in existence in with his mother would not increase the threat to women) and undermine their sense of privacy and dignity.
Another thing people struggle with there is that the focus of the analysis would have to be what benefits it brings to women, not the sub-section of men who want to be included. Not a natural formulation for many people to centre women!
I suppose you could also have a crack at arguing that allowing young boys in with their mothers constitutes direct discrimination against other, older males on the basis of age. I'm a bit too knackered to consider and set out the legal reasons why that would be likely to fail but I think the judge would also resent having to write anything more than "oh fuck offffff" for his or her legal reasoning!