Op, I could have written your post word for word, except that I've moved further on in that I've totally cut contact with him.
I was never allowed to disagree with him - mum would always say 'don't go upsetting your dad' or 'your dad's always right you know'! He was a total bully. He made mum cry. He drove me into the floor telling me my life was a pity and a shame. That I was useless and stupid, and that if only I'd done ABC I would have Achieved XYZ.
So, he says that HE is going to live with you. What about your mum then? Or is this if he ends up alone? I secretly thought for years that like you, I'd have mum live with me in a heartbeat, but not him, never him.
You really need to do what PPS have suggested and say 'that's not possible' EVERY TIME, then change the subject, walk away, whatever.
I've paid the price by not being able to see my mum anymore. I can't walk into the house as they have large locked gates that I can't climb over. If I tried to phone mum and he answered he just put the phone down on me, so I've given up trying.
I'm a few years down the line now and honestly, even though I miss my mum badly it was still the best decision for me. No more guilt tripping, I've picked my self esteem up off the floor and am now successful in my career. I am living my life. Not his.
Make a decision in your mind to be ruthless for the sake of your own life, family and well being - to mirror his own behaviour. 'No' is a complete sentence, let him sort himself out - oh and stuff his money. I had the same thing said to me -that I would be cut out of the will if I didn't do as he said. Well whoopee, I didn't do what he said and the sky didn't fall on my head. . The first time you say NO is the hardest, but then gets easier.
We are trained from birth to obey the patriarchy - It's time to rebel methinks. Stand firm.