I think the fact people focus so much on immigration from Muslim countries rather than from the EU (especially pre-Brexit) proves that Islamophobia and racism are at the heart of many people's objections.
Re the housing crisis - it's horrific, and, I'm personally affected, as a non-homeowner. But, once again, as with so many issues, immigration is being scapegoated for government failure (successive governments dating back decades). The selling off of council homes by Margaret Thatcher is now having a huge impact on housing shortages. The failure of several governments in a row now to introduce proper rent controls, and other legislation to keep both rents and house prices down. Among other things. Many politicians are part of the landlord class, and they don't care about the homeless and vulnerably housed. But, they don't want to admit that, and it's easier to blame immigrants, so that's what's happened. Very unfair and totally predictable unfortunately. Throughout history, minorities have been 'othered' and blamed to cover the cracks of government and societal failure.