I desperately want to say yabu, but after reading both sides of the press about this every day... I don't think you are 
It's not immigration that's the problem, it's lack of integration. And you can't tell from the outside who is one of the 99% of genuine people, and who is in the 1% of twats who hates the west (but still wants to live here because the things they hate are the very things that make it a desirable place to live).
I'm a lawyer. I went to a meeting with a friend the other day because of a legal issue he is having. The other party doesn't speak english (unless it suits him, as he piped up halfway through, much to my surprise!!), so he brought representation, the chair of his local mosque. Who said pretty much immediately: "we all know what women are like - too emotional and irrational to make sense." And this was a pleasant, well educated man, a leader in his community.
Views like that and more dangerous ones are very much a minority at the moment, but I do worry for my DD that it won't stay that way if millions of people migrate here without the time to assimilate to our way of life. In fact, my own tiny personal experiences of racism have all been from women (eg I've been called a white slag and pushed into a busy road; I've been refused a seat in the public area at Westfield when 8 months pregnant) and I do think that is an issue too. What are these women hearing all their lives if they think it's ok to despise another woman for wearing a short sleeved t-shirt? It's never bothered me for myself, as I can see that it's a tiny part of life and that the huge majority of other cultures don't act like that in any way - but I think we worry much more about our children than we do about ourselves!
Of course it cuts both ways. Any britain first type who has been hostile just because of someone's appearance or background is also v culpable in terms of preventing integration and harmony. Make people feel isolated and resentful, and you have fertile ground for more extreme views.
I also think that the pressures on an small country in terms of housing, schooling, hospitals etc will lead to more tensions and isolation if millions of people (which it could well be) arrive here, and that could increase poor behaviour on every side.