I think the issue with privilege, is that the people benefiting from it(whichever it might be) are afraid something might be taken from them. They don't quite know what, definitely not the privilege as they have none, but something. The defensiveness and whataboutery shows that even better. There's a reason why the main argument is about someone who has nothing or has very little.
I'm white and I have privilege, even though I'm non British.
For a while I lived in a really shit,racist estate. Black people were harassed, shouted at,stuff thrown at them or worse etc.
I've had more than a few comments myself, but unless I opened my mouth(and made my very obvious foreign accent known) no one would know.
Black lady on a bike got a can of beer thrown at her and "go back to your fucking home" shouted.
When I went past ,no comment because I'm white. I passed.
Does it matter if that lady is a bank CEO and I'm a cleaner? No .Does it take anything from me or my life to accept that I am privileged due to my skin when it comes to certain situations/circumstances? No.
Accepting that you might benefit even if involuntarily from an advantage does not take anything away from you,your experiences,your life, your struggles.