I'd have thought someone who deliberately gets in for a swim is at far less risk of drowning than some who falls over board, slips on a muddy bank, is pushed and overbalances when larking about. Plus if it happens in the dark, there may be no one to see to be able to help.
Open water swimmers usually have a good idea of how to judge water conditions such as the dangers of deep quarry pools, rip tides, etc, and know their own tolerance to cold. I've certainly chosen not to go in or stay in because of strong waves or currents or getting cold. Open water swimmers will be dressed for it in a costume or wetsuit, not heavy outdoor clothes that will weigh you down and hamper your movements.
You're less likely to suffer cold water shock by getting in in a controlled way than falling in unexpectedly, and gasping and breathing in water.
Hypothermia will also hamper your ability to judge the danger of the situation and take action, e.g. swim to shore, and cold stiffens up your muscles (because it sends the blood to your core to protect your vital organs) and you physically can't move as easily.
There are quite a few reasons why it's more likely to happen to people falling in rather than planned swims by experienced open water swimmers.