DH and I are raising our son (now 19mo) bilingual (English/Serbo-Croat), with both of us sticking to our native tongue when speaking to him, but speaking to one another in English. Most of DS's words, unsurprisingly, are in English (unsurprisingly because we live in an English-speaking environment, and his mother tongue is English, whereas his "father tongue" is the different one!).
DS hasn't seemed particularly concerned so far that his father speaks English to me, and various friends of mine have managed to get the children to speak the father's language (as opposed to just understanding it) by the father's speaking purely [whatever the language] to the children.
I occasionally have to resort to the formula: "That's right, darling. Tata says 'pas' and Mummy says 'dog'." Not sure how this will work for more complex grammar, but we are trying to set the idea of difference from the start. Unfortunately, my MIL mixes in a lot of English words when speaking to DS, which we are all (DH, FIL, and I - poor woman is surrounded!) trying to get her to stop doing. When she does that, I try to use the formula again: "'Pas'. Baka says 'pas' and Mummy says 'dog'."
A Polish father (his DW is British) we know had to repeat "What was that?" in Polish until DS1 replied in Polish. DS1 has a rather inhibited personality, so needed the extra push (think of it as gentle discipline, or coaching the child on how to eat with knife and fork), but now speaks it quite happily, even unconsciously, with his father. That family has now moved abroad, and DS2 (who is less inhibited than his brother anyway) apparently said his first recognisable word in French (from nursery/garderie, I suppose).
Children do seem to be able to sort things out eventually, and it can take longer for them to start speaking, but it is said to help them intellectually, since "more than one language" is a very abstract idea, and once a child grasps something as abstract as that, s/he will hopefully be more open-minded to other ideas.
Good luck with your efforts! It's a really worthwhile thing to do, and your DS will be a bigger person as a result , and will be able to communicate with all his relatives and family friends on both sides. If you ever get really worried, you could drop back to 2 languages, and re-introduce Berber later, although I'm not sure how best to manage that; it would be best to take further advice on that....