Non-drivers generally do not kill over a thousand people a year. Drivers do.
Some people just aren't able to drive well enough to do so safely. The reasons for that might be physical or mental. Some people can't see well enough to drive, some people are very anxious about driving. Obviously, people who want to drink should not be driving afterward.
Some people don't have cars. Cars aren't cheap. They cost money to buy, they cost money to insure, they cost money to fuel and maintain, and in some places they cost money to park.
I find it stressful to look for parking in places where it's difficult to find. Here in the US at least, places where it's difficult to find parking are often places where walking is an option. If the weather is good and I don't have to transport anything heavy, I'd rather walk and not deal with the stress of trying to find parking. I do drive, but I like having the option of not doing so. When I visit your country, I don't trust myself to not drive on the "wrong" side of the road, so I don't drive there. (I also think most of your rental cars are stick shift, and I can't drive stick) Those of you who I didn't run over the last time I was there, you're welcome.
Walking is good exercise, and driving is not. I'd much rather walk to some shops than go to the gym.
I'm not much of an outdoors or countryside type. I'd rather walk around in a city, ideally one with coffee shops and book shops. That means driving to places in the middle of nowhere isn't really an issue, because I generally don't want to do that anyway.