Of course they haven't been 'banned for existing'.
By law, (which is what it's always been, the supreme court just confirmed it) if you want to use the legal sex discrimination of saying something is women only, or men only, you can, and exclude all who do not belong to that group. But that means that it is only that group, and you cannot then start permitting some members of the opposite sex in and not others without losing your legal protection. In this case, you cannot exclude some men and not others, it's discrimination.
When you (general you) start messing with this, you take away the boundaries that permit women to have women only refuges (obviously alongside lots of nice accessible options for all men), and women only changing rooms (obviously alongside lots of nice accessible options for men, it's ok, no one wants men to suffer), and women only rape crisis services (obvs men too, they won't suffer by women having accessible provision), and women only HCPs and strip searches and other things. You can't have it all ways. The term 'women only' has to mean what it says. And otherwise women are excluded (and banned for existing, as they have been when they cannot access mixed sex services being called 'womens'). This is inclusion: groups meaning what they say, and services being able to exist along side so that everyone has access equally.
There is nothing to stop anyone announcing it's an activity likely to be of interest to men and vegans and spot welders and nail bar professionals, it's fine. But you can't then invoke any legal right to exclude anyone, and you lay yourself open to legal action. In this case, non trans men who could claim discrimination as some men can do this but they can't, and possibly women who can claim for lack of actual sex based provision when they can't access a mixed sex group labelled for women.