I can see both sides of this as I did make some close friends at work in my twenties but now in mid-fifties, in a male-dominated environment with many younger colleagues, I have less in common with them, plus distance is a factor, so unless there is a lunchtime meal, I rarely go.
Our office is at a central point in NW England and people commute in every direction. So, whilst I come 35 miles from the south into work, a couple of the women whom I do call friends commute a similar distance from the north and east.
I also work 3 days – quite a few social evenings are Thursday or Friday which would mean me travelling on a busy commuter route as I am not in the office those days. I find that I am exhausted enough at end of the week, without adding that on. I also can’t tolerate alcohol as much now, and prefer not to have hangovers, another reason not to go out and drink with them.
Sounds like you are a similar age to me Op – I enjoy my job and colleagues and chat to them at work and if we away on business but I don't feel that being friends with them outside of work is a necessity.
Even the friendships with the women that I mention above have drifted in last 3-4 yrs, partly distance but also being at different points in life. We used to meet up on our days off (we all work PT) but as they now have primary age kids and are tied to school pick-ups, whilst mine are now young adults, it is harder for them.
I try to see them for lunches at work but quite often they want a minimal lunchtime to get away for child care. Having been there, I understand.