It's not necessarily about having lower aspirations; it's a mixture of not knowing what opportunities are open to you (or having a way to find out) and also potentially having far more financial or caring responsibilitie at a young age, so having to turn down other opportunities.
For example - my A level grades might, in theory, have got me into Oxbridge. But back then I had no idea how to apply, didn't know how to navigate the college application system (no internet then to research it on either!), couldn't afford the train tickets to go there for open days, didn't know how to do a personal statement that would've helped an application - etc etc. So I didn't apply and went to a local university instead. I know it's a bit different now, but back then, Oxford or Cambridge may as well have been the moon to people who grew up where I did.
I now know many people who did go through Oxbridge, and they all tend to have had either schools that helped them apply, or parents that did so. Nobody in my family or wider circle had been to university, so as much as they wanted to help, they struggled to. I distinctly remember my mum asking for help from my school in trying to choose a college, and they had no clue. Compare that to a public school where people apply all the time, and teachers possibly went through the system thenselves as well.
Then there's having to work. I had to work part time from being 16, and most people I know did too. That meant not being able to do a lot of the extra curricular stuff, the internships, the work experience etc that helps with a career later on. Working on a till at Tesco doesnt tend to look as good on a CV as having done work placements in the field you're interested in. And of course getting those work placements, especially ones in London, means knowing people in the field, having somewhere to stay, being able to afford the transport and so on.
I'm in a decent job now, and know a lot of people who went through Oxbridge. Lots of them genuinely have no clue that they're privileged to have not had to work during sixth form; to have not had childcare responsibilities for younger siblings at that age; to have had friends and family- or switched on schools - who could get them into decent work placements etc. They often think that if something is free to do, then it's available to all. Which is just not the case, sadly.