To me, it's not rude, I lived abroad for 10 years and was often asked where I came from, it was just asked out of genuine interest.
DD is mixed race, my other DC aren't and I'm often asked in a round about way, " Oh wow, they look so different..." then I have to explain. I don't really mind. Yesterday someone said they had been dying to ask me what ethnicity DD was but thought they may sound rude, I've heard similar things said, but to me it's not rude at all to ask, aibu?