I feel similarly. Another regular swimmer here (I've been swimming at the Ladies' Pond for about 25 years).
In the last few years I've noticed the presence of trans men/transwomen at the pond.
It's likely people who've never been there don't realise what makes it so special. As described above, for many of us it's our happy place, our safe space, where we go in dark times as well as where we experience and celebrate our happiest moments.
The changing room and showers are communal (there are a few cubicles, but usually everyone changes in the meadows or shower room together). We sunbathe topless and can relax (or could) without self-consciousness.
In such an environment, it's absurd to suggest we're "looking at genitals" if we express discomfort at the presence of males. The entire point of the space is to be able to feel comfortable, relaxed, immersed in and connected with our physical experience and sensations.
Personally, I don't feel threatened by trans men/ transwomen in our pond and meadows, but I do possess the basic empathy needed to understand that, for the large proportion of women who've experienced worse abuse by males than I have , it could be uncomfortable, anxiety-inducing or retraumatising.