Cash is a way for those to evade tax, who can.
You could just as easily say that cars are a way for people to get away from bank robberies, using that as justification for banning them; or that nobody should be allowed to cut bread or prepare vegetables because a minority of people use knives to hurt and kill other people. That seems quite a one-sided view, suggesting that individuals probably can't be trusted whereas banks and governments always can!
I'm firmly in the camp of wanting to keep the choice. Of course, modern society necessitates cards and online payments, and they're very often the best/only/most practical way, but there is still a place for cash.
I hate car park machines that force you to pay online, call a number or download an app (often charging you extra for the 'privilege' of paying them). Great as an option for those who prefer it, but I just want to stick a quid in, take a ticket and get on with my day.
Also, as in the OP, if I have coins or notes, I have a guaranteed currency to make payments and I'm not dependent on a bank's systems not going down or having mobile reception. Scammers/muggers can't rob me of cash without my knowing (unless I'm asleep).
I'm not saying that they would/will, but the idea that somebody in a bank (or a hacker) could accidentally (or deliberately) press a button or a computer could glitch and render me unable to pay for/buy anything until it's all (hopefully) resolved, maybe days or weeks later, is not the most comforting thought. Effectively, every time you make a payment digitally, you're asking your bank for their permission to allow you to do so. Although they shouldn't decline it if your account is in credit, they easily can at any time, if they want to.