Governments for the past few decades haven't invested in infrastructure, so the only "green" space available for homes has had to be adjacent to existing towns and cities, i.e. the "urban sprawl" because of proximity to services and amenities (sewerage, fresh water supplies, power, bus routes, roads, healthcare, emergency services, etc).
If they built in the middle of nowhere, i.e. "new towns", the government would have to pay millions (if not billions) to put on that infrastructure. Hence why all the new estates are on the edge of towns/cities!
Yes, to your points re smaller/cheaper homes, but those usually get snapped up by investors/landlords and turned into student or holiday lets. Councils often don't give planning permission for estates of large numbers of smaller/cheaper units due to local complaints re potential disruption, anti social behaviour, pressure on schools and GP surgeries etc due to too many people, hence why they are more likely to give permission for estates of larger homes with fewer people! Councils are also keener to grant "over 55 only" planning permission for town/city centre developments, maybe because they know that local NIMBYs are less likely to whinge about OAP accommodation. That's crazy, because lots of over 55 flats stay on the market for months if not years due to low demand!
Conversion of empty flats above high street shops is problematic as they're usually very expensive to renovate as single units, often with poor or no direct street access etc. You can't have residents having to walk through the shop to get to the stairs to their flat, can you? Shops aren't going to be keen to reduce the width of their shop front to put in a stair case! It's a lot easier above, say, department stores where you can have a single street access for several flats, but lots of High Street stores are effectively single units. We'd need wholesale demolition or redevelopment of entire blocks to make use of all the empty space above shops - maybe with the demise of the High Street, that will happen!
In reality, we need a multi-pronged approach. Landlords/investors need to be taxed till the pips squeak to encourage them to sell their holiday homes to people who want to live in them. Councils need to restrict planning permission for new builds and conversions to prevent them becoming holiday homes (the opposite of what happens in many places now). We need massive incentives for landlords and property investors to sell or renovate town centre empty properties to be converted to residential. Likewise for brownfield sites. Councils instructed to compulsory purchase of empty homes (i.e. where old people have gone into care or have died). But even with all that, we still need millions of additional homes and for that, we do need to build on greenfield sites!