A situation that is really burned on my mind from the last few years speaks to this issue, imo.
I was at work. Usually in the day, we have quite a number of unhoused people. One man in particular, I'll call him George, a man of about 50, was there, as usual, on the computer. He'd been living at the shelter up the road for a few months while trying to find an apartment, which were in very short supply. He wasn't getting many calls, the couple he'd had were from places that were too awful to consider, full of drugs, or vermin, or both.
We also frequently have new arrivals from overseas, be they students or workers (not many refugees where I am) coming in to photocopy documents. And every one of them is staying some place, in an apartment o room, or in some cases in specially built housing for nurses. At least a few a day. Some even work with us, there are three out of about 20 employees who are newcomers within the last 5 years. One in an apartment, two in houses they've bought.
I think you have to be pretty massively lacking in empathy and imagination to not understand what it means to have a shelter that holds 30, as well as a number sleeping rough at any given time, while at the same time people are coming from away who have managed to secure decent housing.
Racists my ass.