I'm sympathetic to asylum seekers. Particularly those - say, from Hong Kong and Ukraine - where we have a shared history/similar cultures. If large numbers of people come to the UK from these countries and don't go back, we are likely to be able to all integrate reasonably happily. The UK's values, including equality for all, one law for all, and separation of church and state, will not need to change for this to occur.
What many people, probably the majority of the UK population, are concerned about are the large numbers of people coming from incompatible patriarchal, sexist and homophobic cultures. Where women are discouraged from working and must cover up, where cousin marriage is the norm, and being gay/lesbian is a serious crime.
Most mumsnetters have an average or higher household income. Many of us own our own homes and/or we have (some) savings, so we have choices. We can afford to move if we truly feel unsafe. We aren't penniless and trapped on a rundown estate in the cheapest part of a struggling town.
I care about the British people - men, women and children - that are the ones most negatively impacted by recent large scale immigration. Who can't afford to move if the very cheap hotel on their road takes in large numbers of unvetted asylum seekers who arrived on a boat from the famously unsafe country of France
That awkward question you mention, is trying to encourage you to worry about those people too. Would you be happy to buy a (large, very attractive and well-priced) house next-door to a hotel accommodating dozens of young, male asylum seekers? If not, why not?