The RC church my dad attends has a fairly significant Pakistani congregation, that plays with people's heads too.
I remember a Muslim teacher asking some Kosovan students about a traditional drink, I said to here, "These girls don't drink they are good Muslim girls" I think it was the first time my colleague realised there are white Muslims.
It's just people's experience, the schemata in their head pockets 'Muslim' in a subset of 'Brown people' and I don't mean that in a bad way.
I had a student tell me I couldn't cook, I responded that I could. She then said, "You can't make Jamaican food" and I responded with, "I make a mean saltfish and ackee"
I could virtually see the cogs turning and 'does not compute' fly around her head.
In my time I have shared a house with Jamaican woman who is white, I have a friend, she is black and her daughter is white as are her grand children. When her daughter was at school and got in to trouble my friend would go to the school and everyone assumed she was the step mother.
@FakingItEasy most Muslims in Britain are darker skinned, but not all. And certainly not all in the world.
OP
I think it is just that we see where we live and the people around us as 'normal' and when someone doesn't fit in that it plays with your head.
I worked with an Indian nurse whose husband was from NI, when she first went to meet her husband's family she was stared at in the street.
Whether you like it or not, people are asking questions because you don't fit their idea. It's curiosity not anything else.