Not everyone just gets in and goes! But you won't see the ones who haven't had the nerve to go out driving.
I barely drove at all for 7 years after I passed, mostly because I couldn't afford to run a car, but also, I made it no priority to be able to afford it. When I finally passed, (4th attempt,) I just burst into tears and said, "I never have to drive again!" My parents were keen for me to learn, because we lived on a farm, no public transport. They would make me drive when I went back home to keep my hand in. I didn't drive so much when I first got a car, because I lived in walking distance of work, and I didn't see any reason to stop walking there - no parking hassle or anything. I did take a couple of refresher lessons, including motorway driving. And it was a lot easier to go and see my parents, and not having to coordinate with them for lifts from the station.
Then I got a job further away, and was driving every day for about an hour each way, and it was doing that which really improved my confidence.
I think some of it was growing older, too - I still occasionally stall, but it really is occasional these days, and when I do - well, 15 years ago, I would have been getting red and flustered and worried about the cars behind me, but now I just shrug my shoulders, put the car in neutral and turn the ignition - any cars behind me can't go anywhere until I've got going again, so if they're getting pissed off, that's their problem, not mine. And as a result, I'm a lot calmer, and tend to get restarted far more quickly (and stall far less in the first place anyway.)
I'm quite a whizz with parallel parking these days, too - this is a result of living in a street where I have had no choice but to practise it every single day. And years of daily practice means I've got good at it. I still occasionally get it wrong, and have to pull out and start again, but the vast majority of times I'm good, and can even do it with an audience. But it's all taken practice - practice, practice, practice. It's the only way to get better at it.