I have said this before, but the general consensus of most economists on immigration is that higher levels of immigration tend to result in a faster increase in wealth overall in the country, BUT the benefits do tend to go disproportionately to those who are better off, and there does seem to be some absolute negative impact on people at the bottom of society (who already have the least, and therefore any impact on them at all is a real problem). And that's just in the short term.
In the long term, high immigration rates are likely to increase inequality still further, because of the effect on what economists call "mutual regard" (=willingness to redistribute wealth in the form of taxation).
So this is about trade-offs. Do we want to increase society's overall wealth to the absolute maximum extent, at the expense of becoming a more polarized and unequal society?
Or would we be OK with increasing the overall wealth of society a little more slowly, but be a more egalitarian and less polarized society?
If the former--keep on with high immigration rates. If we want the latter, we should probably look at gradually dialling immigration levels back to what we had in, say, the mid-1990s. We could still enjoy most of the benefits that immigration begins, but we would be more comfortable with ourselves as a society.
It is not about "good" or "bad"--it is a question of "What sort of society do we want to become?"
By the way, most immigration comes from outside the EU, so even if we stay in the EU (which would make economic sense, quite honestly), we could still reduce immigration levels from the rest of the world if we really wanted to.