It really is his problem here, and you need to make him realise that if he wants to understand what is said between you and the children, he will need to start learning French.
There are so many opportunities to learn french over here. I am sure there are evening classes he can attend.
My husband has had to learn Danish, he still aint fluent, but he tries, and he gets frustrated and wants to quit but he doesn't.
My husband keeps it up by speaking to the children, sometimes me (I am forever going around correcting him, but it is worth it) and he reads the Danish goodnight stories, or other books that we have for the children.
You could get him to watch some of his favourite films in English with French subtitles, and then swap around watching the same film in French with English subtitles. One day he will be able to watch that film solely in French.
However, he knows it is something he needs to do for the sake of the children. I don't think he is bad, and I know he is never going to be fluent. Danish is too hard a language for that ever to happen. Most foreigners who have moved to Denmark don't achieve fluency even after having lived there for 20 years.