This is silly.
First of all, quantity is in play.
Second, people who come from a particular area tend to be a comparable colour.
I would think that the colour of these men is the last thing people have an issue with.
They usually come from very conservative countries whose cultures expect very different things from their women, and allow very different behaviors in their men.
Most of these men will be fine at home in that they conform to the cultural construct of the place.
Transferred here though, that culture will play (for us) as sexist and conservative, with layers of misogyny rolled in.
The simple, easy to see values around women's dress - covered hair or mini skirts for example, versus deeper beliefs around sex before marriage and what that indicates, for them, in the value of a woman.
I do think that the racism accusations around this issue are extremely unfair, even though there will be some - 'cause people 'init?
I have heard many a sneery conversation around this topic in our "leafy suburb."
If bus loads of young men were to arrive tomorrow?
The "racism" would kick in sharpish.