I think some of these religions are already viewed with suspicion by liberal groups about the reasons they segregate in the first place (viewing women as always needing to be protected from men, or dishonoured if they have contact with non familial men) as being misogynistic and oppressive.
I tend to think that there is some truth in that, but regardless of whether we agree with these religious practices, women from these strict religious groups have more right to be in women's public spaces than any man with a self identity problem.