It's a bit chicken and egg.
You have men only sessions. Some women won't come for religious, cultural, modesty, body image, etc. reasons, so will not be able to benefit form the service.
If you audit your service users and see that certain groups aren't using the service, you might set up some specific services to encourage them (e.g. 60+ session, women only, etc.)
You might find that take up is low, high or in the middle. You might find that you attract different people from what you'd hoped. You might want to review your marketing policy to see if there are still barriers or if you aren't pitching it quite right.
Anyhoo, didn't care much for Triumph's comment about "overweight" women being self conscious and not wanting to go to "normal" swimming. Women of all sizes and shapes may feel more comfortable in a single sex session. Similarly, many overweight women don't give a fig about single or mixed sex sessions. Major league assumptions going on there.
Also, I don't swim regularly anymore, but my boss who does says that some men can be very "aggressive" in the pool, taking up lots of space, tutting and cursing if someone gets in their way (particularly women or less proficient swimmers) and although she is over 6 feet tall, she says it's made her feel uncomfortable and she knows other women who've stopped going to her pool (or at least the mixed sessions) because of the behaviour of men.
So, it may not be down to wanting to be in only an all male environment. Women might have no issue with a male attendant or men observing from the restaurant.