Just wanted to add another perspective to the "they learn fast and forget fast": v true, but a child's brain has noooks and crannies where it "keeps" language until needed again . There is research to back this up, please do not ask me to find it (rubbish resarcher, me), but I am happy to share compelling anecdotal evidence:
I was born in Germany to two German parents speaking nothing but German to me. We moved to the US when I was 3 for 3 years. I went to a nursery/kindergarden affiliated to a university. The nursery was used to having to absorb children who spoke no English. I became friends with a little Japanese girl (we obviously "found" each other as we were both adrift) and have been v good at eating with chopsticks ever since .
Anyhow, returned to Germany aged 6, started school, and I remember relatives asking me to say something in English - my mind was a total blank. Fast forward 4 years, I am 10, sent on holiday to stay with a family in the States. Did not understand a work anybody was saying. However a few days into it, I dreamt in English, woke up and my English was back or resuscitated or awakened or summat.
Also, one of the reasons I even tried to teach DSs some German, was to make future language learning easier for them. Again there is evidence that this works.
Like others have said, DH not speaking any German to speak of and being quite in favour of them being bilingual but not actually helping to achieve this (DVDs with a choice of language always on in English, gah!!) is a big disatvantage.
DS1 now informs me that he speaks 4(!) languages: English (yes), German (to some extent), French (oui, non, merci, bonjour, au revoir) and Korean (can count from 1-10 since took up Taek Won Do ) - so at least they are easy to please...
In principle I am carrying on, in practice I am slackening and not proud of it .