Yeah, it's interesting. A friend of mine, who grew up underprivileged and as a black woman in a large, increasingly decrepit American inner-city, married a guy, also black, from a very well off family of academics. She is very bright and struggled through family stuff to get a degree, and has done pretty well for herself. He, however, is a golden boy, and it's like everything he's ever wanted to do has been facilitated, right up to a cushy tenured university position at a very young age, positions on boards, etc. He's totally wedded to the whole ant-racism movement. My friend became less enamored with the movement after he revealed himself to be a cheating twat - not just, I think, because she was bitter at his entitlement, or even because it was so clear he'd been privileged in every way over many other people, but also because she realized the whole thing made her feel pretty miserable and hopeless. It was kind of a scales dropping from eyes moment, not unlike what you see sometimes from women that have fallen for gender ideology.
Our family background over generations, of course, has a lot to do with our current circumstances. Race can be a big factor in how families managed in the past. But it's only one factor, there are all kinds of other things that create generational poverty, lack of stability, lack of cultural capital. There are whole groups of people who have been part of recognized underclasses who are not black, even in the US. In my own family propensity to alcoholism seems to have reared its head again and again, affecting family prospects over generations. And there are other things as well.
You can see similar weird dynamics when sexuality is used as a diversity target in groups, gay men in particular often do really really well in high paying professional spheres, so giving them a leg up in employment over others seems pretty weird.
It's just way too reductive, and as you saw, wealth so often is the real factor that is an advantage - and wealth of your parents, not ancestors 5 generations ago.