I completely understand where you're coming from OP. I am French and have lived in the UK for the last 15 years, I am completely immersed in the British culture and do everything in English. Before DD was born 2 years ago, the only times I spoke French was when I spoke to my family, which wasn't that often as we are not very close and I only visit them once a year. I also lost a lot of words purely from lack of use (and still do now, I have to bring out the dictionary sometimes), and it just 'felt' weird speaking French. HOWEVER there is hope!
The arrival of DD made a huge difference. I knew I wanted her to be bilingual and be able to speak to my family and her cousin so I made a conscious decision to speak to her in French only. In the day it was just her and me so although it felt strange and I was definitely forcing myself at the beginning, it did become a lot more natural as time went on. Her dad does all the English and I speak English with him at home but I speak to her in French and it now has become second nature, although it did take a while!
We bought some French baby books when we were in France for holidays and they are the only ones I read to her (or I try to do an approximate translation or just describe English books). I've also tried to remember French nursery rhymes to sing to her. We got some bilingual toys (I requested that my family give her books, cds and DVDs as presents rather than anything else), she has a bear and a phone that speak in both languages and also the Leapfrog play table. I got a DVD of French nursery rhymes called Les Clipounettes which she really loves, and she knows some of the songs by heart. We also started to hold weekly Skype conversations with my parents, mostly for their benefit at the beginning but now DD is 22 months and starting to talk she recognises them and they sing together and have their little 10 minute meeting every week.
It's clear at the moment that she's using more English than French; if there are two words to choose from she tends to go for the English. It's understandable as all her friends are English, and when I have the radio or TV on, it's in English BUT she understands all the French and some words she only knows in French and uses those. She started to count in both languages as well (both DH and I do it going up and down the stairs with her and she repeats after us). Yesterday she was in the garden looking for snails and when she found some, said 'More escargots!'
French might not be her first language but she will be fluent and there is no reason your baby can't either!