I don't think anyone actually thinks it's racist to express measured opinions about immigration policy. I also don't think anyone actually loses their job simply for expressing measured opinions about immigration policy. And if they did, they could claim for unfair dismissal - political beliefs are protected by law.
Of course, some people seem to be incapable of expressing concerns about immigration without also saying racist things that would put them in breach of their employer's policies or potentially bring their employer into disrepute etc. And if people do express racist views, then yes, they could well lose their jobs. And rightly so.
The solution is obviously to learn to say what you want to say about immigration without actually saying anything racist. I'm not sure why so many people seem to struggle with this. But if you engage with the detail of why people are calling you racist and actually try to understand, you might be able to reflect on how you're expressing your thoughts and learn to express them in a more neutral manner.
I do think it's really important to understand what free speech actually means, as a lot of people seem to believe that it means that people should be free to say whatever they like with no consequences. That isn't quite how it works. We absolutely do have free speech in this country, with certain limitations which are designed to protect the rights of others, e.g. you can't incite violence etc. What we don't have is the freedom to say whatever we like while demanding that our words should have no impact on how other people perceive us, or on how they choose to respond.