For a real life example... I have a part time job where not only am I the youngest in the team (by a long way) but I was also only the second woman who had ever been in the department (the rest all older men, it was generally a job people did closer to retirement.)
I got the job because they had been doing a blind application process, which included an assessed task etc. They liked my submitted tasks so they hired me. My managers openly admitted that they were surprised how young I was, when I started. (They didn't mention the fact I'm female but it was clear that I stood out that way too!)
If I'd been hired on some kind of quota, I wouldn't have had the same respect from my colleagues, and indeed self-respect, I'd have felt imposter syndrome, like my skills weren't enough on their own, even if they were, but we'd never have known if they'd have been enough on their own.
It makes people resentful when you treat people differently based on immutable characteristics. The beneficiary feels patronised and like they aren't innately good enough. The non-beneficiaries feel they've been treated unfairly.
I know I'm getting piled on (why am I bothering? I'll stop bothering after this comment) but the fact remains that a huge proportion of society agree with this view, including some people from ethnic minorities, but are afraid to admit it because it's not right-on. DEI is becoming unpopular. Political parties who openly scorn it, are becoming more popular. Companies are starting to question if it offers value for money.