Personally I agree, but I think this is something where you will see real division. There are a lot of women who don't see the need for groups like this for things like hobbies, and think they are weird, and also dangerous in setting a precedent for segregation rather than inclusion.
And it can be more fraught. Over the years I've been a member of different single sex social associations, though all but one just happened to be single sex, rather than having it mandated. Similar to the WI in that they were centered around hobbies associated more with women. If a man had happened to want to join and do some knitting, it would have changed the dynamic, but then it would have if a woman from a different culture or background had too, given these were all small country places. I don't know that I could have justified excluding someone from the only needlework group in the village because he was a man and preferred that hobby to talking about wood splitters with the men's hobby groups. I similarly remember when I was a kid and many girls were fighting to gain access to boys hockey teams, because their weren't any for girls, their argument was that a) there was no inherent reason for kid's hockey to be sex specific, and b) lack of access. They won that argument too.