I've had this lipoma for about 4 years. It is about twice the size as it was when I first noticed it.
It has been checked by my GP and confirmed to be a benign lipoma so advice was to only remove if it bothers me. Physically I can feel it a little between my legs. I don't think anyone would notice it when I swim as 1) I don't think people are looking at my crotch! and 2) I tend to wear boy shorts. I don't mind if people do notice anyway.
Pic attached. I've included my pants so it's easier to see the position.
I am 54, slim and healthy.