If you swim competitively, then full body shaving may be reasonable.
I reckon at the times I go, most people are swimming without their glasses and can't make out much more than there's another person in the pool.
(This isn't fully logical - after all, I am probably not the only one who has had their eyes lasered, and some will have naturally good eyesight, but even then, I doubt they're staring at me anyway. My swimming is neither so good nor so bad as to be noticeable.)