who have surely never done anything to you
This is the issue really. Members of a royal family, or indeed representatives of any head of state, are essentially ambassadors of their nation. Meghan and Harry have criticised the UK on a world stage - the implied accusations of racism, that it’s not safe, the press is awful, the government is ‘rock bottom’. All those things affect the UK and its people. For example I have been called ‘racist’ online simply because I dare to defend the UK to people intent on denigrating it. I give up now as my lived experience as a POC living here in safety and without discrimination is dismissed. That’s how it affects me.