I don't condone "hostility" or violence but I think this is actually an important point. I think this is an area where people talk past each other and it's important for policy.
In 1991 the British population was 95% white and they didn't even bother asking the question before that. That's a little over 30 years ago, only one generation.
I often see quite frustrating exchanges where people will say "immigrants are changing the culture, I feel like a foreigner in my own city when I get the bus, no-one is speaking English" etc. And then someone will argue about how they actually are second-generation so they don't count, or they were granted asylum so they're legally here so they don't count, or they acquired British citizenship so they don't count, or they went to school here so they don't count, etc.
But the core issue is that there was a huge amount of demographic change in a very, very short period of time (culturally speaking) - in some local areas the effect is even more than at the national level. At a very coarse level, everyone who lives here who isn't white has a relatively recent migration history, even if it's 1-2 generations back. And it's the overall sheer size and suddenness of the demographic shift OVERALL, which was made up of all sorts of visa routes and other pathways over the last 30 years, that people are reacting to. The white British population (excluding other white migrants) as of the 2021 census is 75%. It dropped 20 percentage points in 30 years.
You can have every sympathy in the world for the individuals who choose to come here including the vast majority who came here under legal routes - but if you want to engage with the core issue, you need to acknowledge the scale of the demographic shift as a whole, and acknowledge the cultural implications of it rather than argue about technicalities.