2 generations of non-native language transmission here. I'm bilingual English/French. My mother fell in love with France after living there as an exchange student for a year and is a French teacher. Her French is fluent, a slight accent and very 'correct'.
She brought my siblings and me up speaking French. We all speak French natively with a Parisian accent, I did some of my tertiary studies in Paris and nobody knew I wasn't French unless I told them. My sister lived in France for nearly 10 years and married a Frenchman. They currently live in Germany, she speaks in English, her husband in French and of course the kids' main language is German. Her children speak perfectly fine French and English to my ears and they are doing very well at school.
My partner and I are bringing up our daughter both speaking a non-native language. I speak to them in GÃ idhlig (Scottish Gaelic), my partner in French (not a native speaker but, having lived in Switzerland, fluent). Luckily our daughter has an intense fascination with language and is a huge reader. We subscribe to French cable TV, and I've gone to great expense getting as much GÃ idhlig reading material as possible. And our little girl shares my love of Kathleen MacInnes (the most divine voice ever, so outrageously talented). To hear her singing songs to herself in archaic GÃ idhlig is amazing (and with a perfect South Uist accent, not her usual Harris one).
The big decision came a couple of years ago when she started pre-school. The options were a bilingual French/English school about 40 minutes away or a nearby Japanese school where several of my work colleagues send their children (I also speak Japanese and work for a Japanese company). We were very unsure what to do as, with 3 languages already on the go, we weren't sure what a fourth would do given it would be newly acquired at school. The deciding factors were the proximity of the school, one of her friends was in the year ahead of her at that school and the fact that a chunk of our social circle is Japanese.
Well, I shouldn't have been surprised but she is now a proficient Japanese speaker. Luckily I can help her with her homework and participate in some school activities.
However, we were lucky to have a girl with no learning difficulties, an intense curiosity and interest in languages, insatiable appetite for reading and friends in the various languages who provide social interaction with the languages. She also is also very proud (read boastful) about the fact that she's special in speaking languages that other people can't, particularly GÃ idhlig as so few people speak it, it's such a secret language for her but managing her ego and her tendency to use it as a weapon is something we need to keep a very close eye on.
Despite having 2 of her languages primarily spoken to her by non-natives she doesn't have an English accent when speaking French and my one native-GÃ idhlig speaking friend assures me that she has native accent although she says it sounds a bit like a Harris/Skye hybrid at times.
The main thing I can say is that you need to adapt your situation and expectations to your child's abilities. 15% of children will have language acquisition difficulties, regardless of whether they're monolingual or bilingual. I know of people who've abandoned a bilingual upbringing due to misconceived fears of confusion or language impairment, where the real problem was dyslexia, low self-esteem, peer pressure or some other factor.
I say, go for it. Even if you're language use isn't perfect, your child will be a native speaker and will have the self-correcting abilities of a native speaker. And from my experience kids tend to mimic native speakers over non-natives.