I lane swim, as well as early swims my place does late opening & it's usually near empty, perfect, as not many want a 10pm swim. I've used that slot a decade & love it.
For ours though is a daytime slot that I stumbled on on days off that's even quieter, I've been on my own for several sessions for 30 mins of it which is fab.
It's an old fashioned 1976 pool with almost no updates & having a hot shower is a lottery, so I guess lots of people prefer to go elsewhere as its pretty crap.
However I do still see & v occasionally part of some slow swimmer/fast swimmer kick off.
I'm surprised 6 ft man is considered big? I'm bigger & don't think I'm big. 6ft 8 or so is big.
But anyway to address your point. No its not automatic that women are slower, I'm a fast experienced middle-aged male swimmer, often I'm murdered by younger women, sure they are probably ex competition swimmers but they are rapid. Were 3 of them a couple of wks ago, I slunk off to the medium lane as I was just a roadblock for them.
Last wk was a lady tho approx my age who was comfortably faster than me, we chatted after & we are only a year apart in age so very comparable, she's not an ex competition swimmer, but a recent convert to long distance open water swimming who swims 3 times a wk minimum.
Was a lady posted above who is 17 sec/ 25m, I'm not that fast. Well in a 2 length sprint yes of course, but not for the hour I swim for so, thus she'd murder me, she maybe much younger tho as per the younger ladies I see who are super rapid.