A tenant of Parliament Place in Liverpool told the BBC he came home from work in March 2023 to find a letter from Schloss Roxburghe Holdings (SRH).
Andrew Lewis said he was left in "disbelief" at what the letter said.
The letter told tenants they needed to move out because the building required urgent fire safety works.
Lewis, 47, said: "People were afraid, panicking… and didn't want to be evicted."
The IT worker added: "They put it vague enough to make us assume we could come back in again. It wasn't until someone emailed the management company and said what happens after this four-week period that they said they were going to terminate our leases.
"And that's when we thought, we're not standing for this."
The tenants did not know about plans to move asylum seekers into their homes, nor that the plan had been under discussion for up to three months.
Those the BBC has spoken to also said they did not know that the buy-to-let landlords who owned the flats they lived in had been directly approached by an "agent" of SRH promoting the asylum deal to them.