I'm currently reading 'Weapons of Math Destruction' by Cathy O'Neill
Some of it is terrifying. It looks at big data and how decisions are made about targeting advertising, policing efficiently, how universities in the US are ranked and how CVs are filtered for jobs. It's mainly focused on the US but there are British examples - our data rules are stricter but honestly I don't think they are strict enough.
One of the first things it talked about is how advertisers in certain situations deliberately target vulnerable people to exploit them financially. So if you are in the bottom 40% of the population income wise, they will advertise things like pay day loans to you. Think about this and how certain products (and indeed scams) with unfavorable prices / terms etc are deliberately targeted - and how if you are better off or educated you'll pretty much never see these same adverts. If course this also means those with the most knowledge of how these might be dubious (because of education) never report them because they don't see them - thus they are able to exploit more easily.
A friend of mine was talking about this and how she was currently being massively being targeted by adverts for weight loss drugs. She's a bigger girl and has a complex about it (by her own admission). On principle she doesn't want to try these drugs because she's concerned about the long term unintended consequences and thinks it's too good to be true. But this constant barrage of adverts has made her nearly crack on several occasions. Meanwhile her friends who aren't overweight aren't being picked up by the data and aren't getting targeted by these adverts.
Then the book goes on to talk about how data is being used to filter CVS. This is supposed to remove certain biases but in actual fact a lot of these biases are either inadvertently being programmed with bias in or are being deliberately written to include them in less identifiable ways which aren't transparent and aren't as easy to challenge legally.
It talks about how certain ideas or traits are used as proxies to suggest something else and theres little follow up on whether these are accurate or not accurate.
One example is how the 'bottom performing' teachers were automatically sacked - without thought as to how people were gaming the system and over inflating performance because they got bonuses for higher achievement. Thus a good teacher who came after a cheating teacher got sacked despite having an amazing reputation with colleagues and parents because the data said she'd understand performed.
And how one IT recruitment company had made a connection between star employees and liking a particular type of Japanese Manga so was favouring them (and of course this also massively disadvantaged women in multiple ways). This also begs questions about creating group think on an industrial scale.
The author reflects on these correlations and untested assumptions as akin to phrenology. And she stresses how the actively target and entrench poverty, often by design - and people not subject to this would be completely oblivious whilst those on the recieving end might well be fully aware of how off something was but unable to challenge it because they a) can't prove it, b) need a higher level of proof than all the work going into creating these ideas c) all without the education and financial means to fight back against such injustices - coupled with a culture of non disclosure and lack of transparency making these decision making processes pretty much infallible despite potentially catastrophic flaws which act as a feed back loop without checks to thrive and damage society as a whole.
It's fascinating. The book a few years old now - it was published 9 years ago - but I think we can start to see a lot of the fall out from these patterns. And the trend is towards yet more AI.
The underlying point here is that if you are rich or have the right postcode you are often targeted in a positive way and have more opportunities compared with someone poor or who has the wrong postcode - before you even do a single thing. Working hard is something of a mute point - there are greater barriers at every step of the way. It's crucially primarily an economic thing (race and sex gets caught up in this because of pre-existing economic disadvantages).
It goes a long way to explain many of our current political challenges. If you are on the wrong side of the divide, it's not just that you are disadvantaged - it's that you are also seen as a legitimate target to exploit and/or discount.