It doesn't matter if only one parent speaks the 'non-resident' language, while the other speaks the language of the country they live in, as long as there is consistency. I know plenty of parents who are in the same position as you, where the mum speaks to the children in one language, the dad in another, and they all speak the same language when with locals. It is hugely beneficial to the children, and well worth the effort.
Whatever mistakes the children make in their grammar, vocabulary etc are virtually irrelevant, even if they stop speaking that language during primary school, as they will have learned that there is not only one way of communicating and will find other languages easier to learn as a result.
My db and I spoke Hebrew with our parents, and English with each other and at school, until the family was naturalised when I was about 7 or 8. My parents had come under such pressure from teachers to stop talking Hebrew to us (apparently it was confusing us! ) that we gradually switched over to English and by the time I reached my teens I had forgotten much of my Hebrew. I couldn't read or write it at all any more. But I learned French, Spanish and Latin from scratch at secondary school, and, when I decided to learn French and Hebrew again in my 20s, I was fluent in both within months.
And as for confusing us - what a load of tosh! We were all top or near the top of the class in English, have degrees, and my sister is a journalist with a superb style. I don't think that's a mtter of IQ (db and I lean towards sciences and IT, dsis towards art), but because of our early experience of other languages.