Hi Fazerina,
I don't live in an English speaking country and I am the only speaker of English that my dc have access to. I speak to them only in English. I don't see it as extra work, to me it's perfectly natural to speak to them in my own language. Even if there are other people around, non-English speakers, if I am addressing the dc, I use English. I have lots of books in English that I read to them and if they watch TV, I try to put on a dvd in English for them. Is this an option for you? Can you find children's books/ dvds in your language?
The key, I have found, is consistency. Your ds does not need you to speak English to him, he is getting plenty of exposure to it elsewhere. If you are speaking to him, no matter who else is present, speak in your own language. Don't switch back and forth from English to your own language.
Does your dh speak a different language entirely? If so, does he also speak your language? Or at least understand it? If so, you can speak your language all the time at home, including when speaking to him. He doesn't need to reply in it, but if he understands, then you can use it, and it will give your dc further exposure to it.
I am fluent in the local language, but in front of my children, I use it only when socially necessary, and I switch back immediately to English if I have anything to say to them, even if it's only something like "Stop doing that, ds1" or "No!" Anyone that visits us that also speaks English is encouraged to do so freely with the children. You can encourage your mother and any other family members to speak your language and no English at all with your dc, to help boost it as much as possible.
My eldest is 5 and a half now and he speaks English very well, having used this approach from the beginning. He does make mistakes, usually to do with irregular verbs or plurals (he might say "sheeps" or "I breaked it"), but these are improving, and I hope will get better especially as he starts reading (we're just getting started) and is exposed to more vocabulary via his books.