Given that it's a skill that requires coordination and concentration, it's not surprising that some people find it easier than others (same for playing sport eg tennis - I'm no good at that and not interested in trying to better myself).
But because it is the norm for a person to drive, if you don't, you will miss out on things.
My DH has a licence but won't drive due to anxiety and I must say I find it really irritating. Every time one of the kids needs picking up from somewhere, or when he wants to go and see his dad, it's me who has to stop what I'm doing and head out.
I often wonder what will happen when his dad gets to the stage that he can't get himself to medical appointments etc - will I have to do that to? Or if I got ill, or sustained an injury that meant I couldn't drive, how would we manage as a family? Or if we split up; there are a lot of places it would be hard for him to visit in future.
There is public transport where we live but when the kids are coming home late from somewhere it's much easier and safer to be able to hop in the car to fetch them.
Being able to drive is just a life skill that opens doors, and makes life easier and more fun.