Exactly. It doesn’t justify letting in hundreds of thousands of young men, many of them violent criminals. Nor does it justify exploiting the student visa system.
When immigration occurs on a small scale, and you are highly selective, it really can benefit a country. After all, T. S. Eliot, Henry James, Joseph Conrad, Freddie Mercury, Clive James and Sigmund Freud were all immigrants. But an illiterate, unskilled drug dealer does NOT enrich your country, and anyone who says he does is a brainwashed lunatic.
Also, when those who come have an affection for your culture (loving Jane Austen and Monty Python, etc), you can maintain a sense of identity. However, when the people you let in either know nothing about your country or, frankly, hate it, then you’ve got a problem. Immigration has destroyed my identity. I no longer live in a country with a shared history and shared culture.