The xenophobes were small enough in number to easily be ignored.
It was the vicious accusations of racism and xenophobia which made conversations about immigration impossible. MN offers plenty of evidence of that.
UKIP all but disappeared after the referendum and now because nothing has changed wrt to immigration we have Reform. Some people seem to think they will implode before the next election, they could be right, but be careful what you wish for, my prediction is that if they do implode then what will rise from the ashes will be far more extreme.
Where are we now? The populist right rising in popularity all over Europe. It's a very predictable reaction to cancel culture, the writing was on the wall at least 13 years ago but it didn't stop the insults coming from the "tolerant and inclusive left". You reap what you sow but sadly so will the rest of us too.
Policing conversations about immigration seems to have been a massive own goal so far but feel free to keep on doing the same thing and expecting a different result.