Why would it make everyone poorer? We just need to spread things out again, how it always used to be. Big organisations/employers would still make the same profits if they moved all or some of their staff to regional offices. After all, their customers are spread all over the country, especially the likes of banks, insurance firms, professional firms, etc. They no longer need to be close to each other, like in the old days of banks and solicitors having "walkers" to hand deliver documents between themselves!
In my home town (Northern run down town now), we used to have regional offices for two huge insurance firms - both were closed down in the 90s upon centralisation to London with the loss of thousands of jobs. Likewise we had small/local offices of national chains of accountants, solicitors, banks, etc., - all now long gone, with the centralisation in London, again, hundreds of jobs lost.
No decent local jobs means graduates never return from Uni as there are no decent jobs for them, and often, no decent jobs in their Uni cities either, so they end up in London/SE because that's where the jobs are!
Basically, London and the SE has sucked the life out of regional economies, aided and abbeted by London Centric governments. It needs a reversal and that needs to start at the top, in Government. We desperately need to get business and enterprise back into the regions, we need "localism" policies to regenerate all the run down towns.