I don't think NOT getting into an Oxbridge college will end your opportunities in life, but going is a serious leg up. I know I was offered interviews/work experience on the strength of my very ordinary Oxford 2.1.
Far more importantly, it gave me confidence in myself while forcing me to shed a lot of completely unjustified self-belief. I was very arrogant having been an extremely bright student throughout school (like, national prize-winning level). At Oxford I was absolutely average and I had to work like mad to stay at that level. It taught me not to be lazy. It taught me to argue and to think. I was surrounded by clever, confident people who were, almost all, kind and understanding and a bit nerdy about their own subject. It was a bit best-of-times, worst-of-times because it was so very intense, but it was formative in a way that my masters at a highly regarded European university was not.
I went to a small but wealthy college and it was the most wonderful experience with minus numbers of hooray henrys and a great mix of students. I'm really sorry for anyone who hated Oxford but I must stress it is a tiny, tiny minority of students who don't thrive there. Oxford isn't interested in aggregating results to show what a good university it is; the pressure is mostly self-imposed. Not everyone likes the tutorial system (although it's a lot less stressful and fairer than seminars, where one male student will often hog the whole discussion and shy people are at a disadvantage. The facilities are amazing. The college system offers incredible peer support through the JCR as well as college 'parenting' schemes and social opportunities. You have contact with tutors in a way that will highlight your issues and weaknesses - which may not be comfortable, but stops people from getting lost in a big university environment. The college system means that so much depends on your college and how it tackles issues for students who need more support either through disability or through mental health problems or issues with assault. Most of them are extremely aware and focused on welfare of all kinds. On a recent visit I found my college had adapted ground floor rooms and bathrooms for disabled use and equipped a library area with extensive resources for disabled users. One of my best friends there was severely dyslexic and had extra support (and this was twenty years ago).
I'd love to see more BAME students applying to Oxford. There were only a handful at my college (including two of the most confident, successful men I've ever met). It is a place where BAME students can thrive, but it seems self-perpetuating that those young people won't feel comfortable applying to somewhere they aren't sure they'll be welcome. The whole Rhodes scandal hasn't helped. My own view is that the university has a very, very long history and many of its richest endowments came from quite appalling people all the way back to Elizabethan times and earlier. I don't imagine Rhodes intended his scholarships to go to BAME students; all the more reason to take them up, I'd have said, and wear the name with pride.