I'm in Japan - it IS difficult isn't it. I'm afraid I overcompensated for being very obviously NOT Japanese (no blending in here, however well you speak the language!)
I speak (and read) Japanese fluently and with a native-like accent (or so I'm told), so I wanted to "show" people that I could indeed speak the language, however foreign I looked.
I also immersed myself in my local community, going to only local playgroups etc. so I just tended to speak Japanese most of the time.
And of course, trips back to the UK are only every two or three years...when we do go back, the DCs are actually very good at communicating with family. And there is Skype, for more practice.
I did try to speak English only at home, and I read to the DCs in English A LOT, so they do actually have a receptive understanding.
They are 11 and 12, and are actually speaking more and more English. There was a time when they were very anti-English (didn't want to be "different" in front of their friends etc. and I didn't want to push it, with my being a big blue-eyed furriner to boot!)
Keep speaking as much as poss, I have days when I regret not being stricter with myself...ho hum. Well, at least they will learn English at school and uni here