I'm a native English speaker, living in France with a French DH. I work FT, so my DC's access to English is limited and I try to make the most of any opportunity I get to speak English to them.
DS2 (3) is not, from all the evidence available, a particularly sensitive child.
DS1 (6) is.
However, when I speak French to them in order to be polite to my MIL, who has complained that she doesn't know what I'm saying to them, DS2 gets very cross and tells me to stop speaking French, because "Mummy speaks English."
He knows damn well I speak French, he hears me speaking it every day to DH and everyone around us except my DC. But he is adamant that that is not our language.
DS1 couldn't give a stuff.
I think it's because the 3yr old is still sorting it all out in his head, and doesn't like confusion, whereas the 6yr old has passed that stage.
Fortunately, my friends here are all fine with me speaking my mother tongue to my DC. If they weren't, I hope they'd find a tactful way to help us solve the issue.