Poor you! It sounds more than stressful. I have some experience of what you are feeling although not to the same degree. (I have a love/hate relationship with where we live!) But can totally sympathise aobut giving up your career. You have given it a good go, but it is not working for you, emotionally or financially, so something needs to be done.
Your dh doesn't want to live in the UK. And you don't want to live in France. It's a very tough situation but bluntly, one of you must compromise. It sounds horrible to say it, but I would have thought that your current financial situation situation gives you the upper hand here.
Tbh, if he doesn't earn enough to support you all, then you have to work, and so you have to move where that can be achieved.
And he must realise that it is not a sustainable or desirable option to live on money you have inherited from your family? Quite apart from the fact that you are spending your savings, what does he intend to do when the finances run out?
(I don't mean to make him sound like some sort of ogre, but if you don't work, and then you have no savings left, you are being put in a very vulnerable situation.)
What are the reasons your dh has for not wanting to live in the UK?
It's very positive that he speaks fluent English. Does he have other transferable skills? Do you have a specific area of the UK in mind that you would like to return to? Could you start putting out job feelers for him there and then see if he is interested? Perhaps once the situation is more concrete and he could visualise a job in the UK, where he would be working and living and where the dc would be going to school, then it would become a more viable option for him (instead of just a blanket "going to live in UK" idea without any specifics).
Good luck !