if they were watching him and dived in instantly, I wouldn't worry too much; a wetting won't have hurt him
(but then we are quite a seaside/boating family, no member of mine who hasn't been fished out of water at regular intervals and we're all fine)
if he was left unsupervised so they didn't notice he was near water and he might potentially have drowned before they did, then I would make sure I was around during visits to keep an eye
but I wouldn't deprive them on visits, hardly fair on your ds to take away his grandparents when he's done nothing wrong
I would also be a bit wary of refusing to see people because your ds has a slight accident in their charge (unless they have really been criminally negligent)
one day, he is quite likely to have an accident when in your charge: will you stop seeing him then?
I thought accidents was something caused by other people's carelessness and that my dd would be safe with me
until the day I stood looking down on her unconscious little body and realise that accidents do happen and a mother's presence is no guarantee that they won't