Because it's lots of extra work to attempt to get in - not worth sacrificing sixth form social life if you're not fairly confident of a chance, very determined and very committed to your subject, or perhaps more importantly, to the notion of going to Oxbridge.
It's not the best place for many subjects.
It's more work and more intense than other places. Maybe that's great if total absorption and intensity is what you want. But, a very bright student might be likely to get a 2:1 at Oxbridge or a first elsewhere, because classifications are relative to your cohort. Lots of people would prefer the first.
I think confidence and competitiveness are key. Lots of bright people aren't driven by competition or status. My experience is that many Oxbridge graduates are.
It is also that quite a few are 'narrow over-achievers', relatively immature and unworldly, not the all round bright, wise, engaged people I somehow expected them to be, probably quite naively. (Yes I said some, of those I've met). This perhaps reflects a particular focus that some entrants have, that many people who aren't so bothered by the cachet don't and, they can have the breadth of awareness and perhaps wisdom to see all sorts of wider advantages to going elsewhere, that suit them, as an individual.