In a nutshell: your colleagues were victims to normalcy bias.
They simply never considered that circumstances could or would change to their detriment. It is a phenomenon that seems very particular to Western developed countries and is particularly acute in Britain.
I come from a very different cultural and ethnic background, and the extent to which most Brits and North-Western Europeans assume continuity and stability in this manner astounds me.
For example, Sparky's comment: There'll be no mass deportations, of course.
You simply do not know this. In ten years, the political and economic situation in Britain and the EU could mean Britain institutes a policy that means the deportation of certain European nationalities, regardless of whether we see Brexit or not.
If you think this is unrealistic, then consider what may happen if the situation between the EU and Russia continues to deteriorate, and which EU countries may feel Russia is a better horse to back -- particularly in light of supply from GAZPROM or a series of no-strings loans from Russia during a time of austerity.
Throw a spark into the situation that leads to an open declaration of conflict and ... you could find that nationals from Greece, the Balkans or any of the Baltic states suddenly constitute a national security risk.
Shit changes. And when it does, it can change fast.
War with Germany was completely unthinkable to most educated British people in 1905. Back in the 1980s, a lot of people that lived in the urban centres of Yugoslavia, as it was then, didn't even know they were "Serb" or "Croat" or "Bosnian", let alone what their neighbours were. In the early 1970s, Cyprus was not a divided island.
I mean, come on, Britain nearly had a military coup in the mid 1970s.
Things can change fast.