We do minority language at home (live in Germany, DH is German but genuinely bilingual - native English speakers rarely realise he's German/ not a native English speaker unless told) and my kids have all stayed home with me til the month of their 3rd birthdays, speaking only English with me, and then started German state Kindergarten (mornings only, 20 hours a week) at 3.
My older 2 kids were both indistinguishable from monolingual German children by the time they started school at 6, according to everybody including their teachers - DD's German grades are well above average (every other child in her primary school class was a native German speaker) and her school English (as a foreign language) teacher refused to believe she was an English native speaker initially, being familiar with DD around the school but not having taught her English - she believed the class were playing a trick saying there was a native English speaker in the class until DD managed to force herself to speak in English to the teacher - it is quite a mental leap for a child to reply in a language other than the one they are being addressed in!
So minority language at home works perfectly! Thankfully the head of our wonderful Kindergarten, despite having never had a non native speaker in the Kindergarten before, was absolutely supportive and never once tried to suggest I speak German to the kids - and my eldest does have a natural gift for language so paved the way for her brothers who followed her path. Mind my German was so bad when DD started Kindergarten nobody in their right mind would have suggested it being used as the main form of lanuage between me and my child (a few xenophobic local elderly busybodies tried telling me I lived in Germany and should now speak German absolutely all the time, and always with my now German children, but they were easily ignored, and as it turned out those bods actually don't and can't speak proper German themselves, but a dialect...).
My only caveat is that we are absolutely immersed in German outside our front door, and pretty much never socialise with English speakers except when we go to the UK - we tried to for a while, but we live very rurally and its logistically impossible to get into the city where there is a native English speaking community in term time. This means my children have picked up German at a functional level before starting any formal education - just in the playground, at with-parent toddlers groups and music groups, and out and about, and have had native German speaking friends before starting Kindergarten.
I think the only thing you need to change is the exclusive socialising with English speakers - invite lots of potential new friends over as much as you can, and hopefully he'll be coping easily in French within 6 months and truly bilingual within 2 or 3 years. Its the mixing outside school that is key I think.
If you speak French to him he will eventually stop speaking English, unless your DH spends a lot more time with him and fully engaged in talking to him than most dads sadly are able to...