snowmoon, keep up the cantonese! It doesn't matter if you are the only speaker. I have friends here who are German /Australian family, living in Russian community, the kids only hear german from their mum, and that is their strongest language. You only need one regular speaker to interact with the child and they will pick it up. (we have aRussian babysitter one afternoon per week, and as a result he understands Russian)
My mum phoned me last week to tell me about new research that shows that kids who learn a second language before the age of 5 are brighter than monolingual kids. This is because the second language helps them to develop extra connections in the brain, and these benefit all round intelligence!! Now I don't know if it is true, but it is a great reason to give to anyone who is trying to stop you from doing it.
Austriamum, we used to have lots of kids come in to school (in England) aged 5 with no English, and after about 2-3 years they were fluent, so if they are going to go to school they will get it eventually. If you can persevere with some kind of playgroup/kindergarten, creche, or anything where the kids are together for the younger ones that will help. Also, although local kids only play with each other, could you invite some round together? Once they have played together a few times they will get used to the idea that your kids don't understand everything, but want to play. I think it is easiest for younger kids, and hardest for school age kids. We used to reckon that it took a whole year for a kid to settle in school if they had no language, most of that time was spent "listening" to the language, and then they would begin to participate. A lot depends on the school, and if they ever have other kids in who don't have German. If not then you might need to be a bit more pushy about asking them to help your kids settle. (for example, I had a kid come in who had a bit of French, no English and then his mother tongue was a local dialect, he really struggle to settle, until I found a handful of french words that he knew, and they just seemed to make a bridge for him. He then settled, and went on to learn English)
Hope that helps