I think it depends on how the concerns are expressed.
Having concerns about immigration does not automatically make you racist. I know people who are concerned about immigration who I do not consider to be racist. But a lot of people who have concerns about immigration express those concerns in very racist rhetoric. That understandably leads to the conclusion that they are racist. If they were not racist, they would find a better way to express their concerns.
In the same way, having concerns about the Israeli government does not automatically make you antisemitic. But a lot of people express these concerns in very antisemitic rhetoric. IMO, that makes them antisemitic and therefore racist. If they were not antisemitic, they would find a better way to express their concerns.
We have to be able to have conversations about the pros and cons of immigration, and it cannot be taboo to express concerns about the way in which it is managed. I don't think people are racist for raising such concerns unless they express themselves in a racist manner.
We have to be able to criticise the Israeli government, just as we might criticise the US government, the Iranian government or the Russian government. Israel should not be held to higher standards than anyone else, but equally, it should not be immune from criticism either.
The problem arises when people use issues around immigration or the Israeli government's actions in Gaza as vehicles for the expression of their hate towards particular groups. The issues just become an excuse.