I have similar thought processes when I give things up (though not exactly the same). My ruminations don't usually focus on the practical consequences of giving an activity up (such as your imagining scenarios in which you drown because you haven't learnt to swim). Instead, I tell myself in a horrible cruel overbearing narrative This is what you always do. You start something and you can't stick with it. You are a quitter, irresponsible, pathetic, unconscientious, flaky, awful, useless.
I wonder if that is also a strong strand in your ruminations, and the thoughts about being punished by drowning are a riff on this basic self-cruelty?
I've learnt over the years that I have to actively prepare how to deal with this voice when I consider giving something up. I retired fairly recently, and because I am being treated for depression I saw the 'social prescriber' at my GP surgery. I've started three new activities, and I keep on reminding myself that this stage in life is one of transition and exploration. It has to be approached with openness, flexibility and self-compassion -- which means giving up on the things that aren't bringing you joy and trying something new (or just taking a breather).
I think that coming to activities via the social prescriber helped me to approach things with a compassionate awareness of my limitations, and the challenges that depression brings. If you don't have the chance to use a social prescriber, could you perhaps be your own 'social prescriber', in the sense of speaking to yourself in the voice of someone who is informed and caring in relation to the difficulties that you have?
Over the years, my 'cruel voice' has made me stick with things that I should have abandoned earlier. That is just as damaging as giving things up too soon. Trust the kinder narrfatives in your head and allow yourself to acknowledge that giving up is sometimes good.
Edited to add: You joined the class because you thought it would be fun, not to keep yourself safe from drowning. Turns out that it wasn't fun for you. And fun is important. It's not frivolous to seek it out, and then turn away when an activity doesn't bring the fun you expected. You're allowed to to that. 