Downsizing, or whatever you want to call it (in my case, financial ruin) isn't actually as scary as you think it will be. I had a very well paid job and my own house. I could afford nice things, holidays etc. Then I was made redundant, then I fell pregnant, then I split with my partner and ended up, two years later, forced into bankruptcy and losing my home. At the same time, my DD became very ill (I'd walk out of PICU to find lots of threatening messages on my phone from the mortgage company) and my Dad died. It put it all into perspective. I told the mortgage company to take the house back. The bankruptcy freed me from debt and gave me a new way of living (i.e. within my means, not such a bad thing).
A further two years on and DD and I live in a lovely rented house. No, I don't own it, but, the way I see it, I never owned my previous house, the mortgage company did, and I have more security as a tenant than as a homeowner. I work part-time in a job I am ridiculously over-qualified for but which I love, and I am now in the second year of a degree course. My point is, don't let bricks and mortar (which, technically, you don't even own, and probably won't for many, many years) influence your life choices. Identify what will make you, and your family, happy, and cut out all the things that are stopping you reaching that goal - job you hate, financial commitments etc. It will be painful and scary in the short term, no doubt about that, but the long term gains will be worth it.