The question, actually, is "could you have been happier without him?" and "did you limit your own life chances because of him?"
Pithy, with respect you're taking too much responsibility here and that's a bad thing. You can and should only take responsibility for yourself and for your son.
His choices were his responsibility.
You can acknowledge your actions had a consequence on him - mainly that consequence was that you gave him a live-in servant and nanny with a shag at times. The consequence for him of you standing up with more self-respect than that was that he lost all those things, and his choice of actions was to berate, manipulate, hurt, pressure and send you ultimatums and vicious words.
Did he value you? Did he speak to you with respect, at all times? Did he cherish you?
If the answer to any of those are No, then that is his responsibility. Has he ever, once, shown any genuine signs of taking it?
Please stop thinking that everyone is reasonable. Some people aren't. Some people will take, and take, and drain you dry and then leave you empty and shattered and THEN blame you. It's right for you to guard yourself against that, and wrong for you to take responsibility that is not yours.
Keep hearing X’s words said in bitterness: “all you care about is money and keeping your house nice so that’s all you’ll have, money and a big lonely empty house with no one who loves you” One day I hope you'll stop putting the Ex's words ahead of your own words in the silence of your own skull.