OP, being asked where you are from isn't in the least bit racist. It's just normal conversation. You two sound like hard work.
Being curious about difference isn't racist.
You are not from here. 90% of the people you will interact with on a daily basis will be from here. So it's natural for people to be curious when they speak to someone who doesn't fit that demographic.
I lived in another country for a number of years, people there always asked me about my history, it was just a normal conversation. Think of it like this . .
Random Stranger
"Where are you from?"
You
"Oh my God! You're so racist! how dare you ask me where I'm from! You should be ashamed of yourself, get out of my house you racist!"
Now consider this interaction . .
Random Stranger
"Where are you from?"
You
"Well I grew up in Mauritius, but I've been here for 20 years now."
Random Stranger
"Ah, I thought I recognised the accent, my daughter is married to a guy from Port Louis, she met him when she was on a study programme over there, she's a biologist, they live in London now. Actually his Dad was originally from the Diego Garcia, so his was one of the families given British Citizenship after the illegal clearing of the islands, he came here when he was seven or eight. Lovely guy, he's a GP. We go to Mauritius every year now though, stay with the family, they live in Flic en Flack now"
You
Wow, small world isn't it. Flac is lovely, my aunt lives there. He was so lucky to get that citizenship, only a few people have it that way. We had to go the regular route with visas and stuff, cost a bloody fortune! Speaking of which, any chance of a little discount? Fellow honorary Mauritian! Go on! You know you want to!"
Random Stranger
"Well fair enough, OK, I can do 15% but that's as low as I can go, I have plane tickets to pay for!"
See how a little conversation can be a lovely thing, and maybe save you some money.
Stop finding offence in everything OP.