If you decide to move, and I think it sounds like you shouldn't, have a very clear idea of what you want/need wherever you decide to move to.
We moved mainly for family reasons (otherwise would have loved to stay in London) but we had conditions about where we moved to.
We definitely didn't want to be in the money obsessed areas (Weybridge, Esher, etc.). Walking distance to a mainline station, shops and pubs.
We have ended up in a lovely big village, with a very friendly and active community. Voted Remain, mainly Lib Dem councillors 5 minutes to the station (just over 30 minutes to Waterloo). On the way we pass a Co-op, greengrocer, fishmonger, butcher, bakery, newsagent, etc.. 10 minutes or less to walk to 4 pubs. So you can do it without losing too much.
But yes, it sounds to me like you should stay!