There is an infrastructure problem and, I'd suggest, a social policy problem. England - not the UK - has a high population density of approx 450 people per square km. That's more than Japan, less than the Netherlands.
While we have many underpopulated regions, these tend to be poorly connected to London and the rest of the UK (I live in one). Their populations are older and their economies declining.
These are, rightly, being targeted for new housing - against vigorous protest - but there's insufficient provision of medical services, schools, shops, social hubs and transport. There are no jobs, which is why younger people have been moving away for decades, and the lack of economic activity leaves regional authorities too poor to undertake increased service provision. Slow road and rail links discourage new enterprise, so there won't be new jobs for the new people.
This could all be sorted with centrally planned and funded policies. But we have become wary of government intervention. It'd be very hard to get the country's co-operation. Additionally, increased home ownership and other changes hamper social management. Back in the mid-20th century, the government (both parties) quietly operated housing policies designed to ensure that areas would be mixed. If they could even do that now without being hung out for discrimination, there isn't enough social housing to have a general effect.
I agree with comments above about shambolic immigration and asylum policies. I'm saying numbers and infrastructure are a problem, too, and this is (broadly) why.