It IS unusual. Not saying it's bad, but unusual. However, it's up to you, if you don't want to travel - don't - it's not compulsory. And nothing to be ashamed of.
However, what really really baffles me (and I mean genuinely!) is that OP and some other posters don't have a passport! OK, I'm an EU citizen living here, so I probably don't know/understand some things. But how do you go about your life without a passport? Do Brits have some other acceptable form of ID? Driver's licence? But in order to get a driver's licence, you need a passport first, no?
Just from the top of my head in the last few years I needed my passport to:
Rent a house (to show the landlord, he made copies)
Register with a GP
Open a bank account
Get a business loan (I'm sure it would be needed for any type of loan)
Buy alcohol (I'm 32 and look pretty much 32, not 15, but I still get asked sometimes)
Fly abroad (ok, this one doesn't count in OPs case)
Apply for a single occupant council tax discount
I work for myself, but I think a passport would be needed when seeking employment?
Apply for uni
It's all I can remember now, but I could think of more, if needed. So what could you use in these cases instead of passport? Birth certificate? I'm not trying to be snarky here, I'm genuinely surprised.