I see this thinking a lot - "don't blame the lack of whatever in the UK on migrants."
That's a misunderstanding. No one is blaming migrants for the fact that, for example, people in the UK won't do farm work. Migrants are behaving totally rationally, from their own perspective, but that doesn't mean that is the kind of policy that is good for the nation, economically or otherwise.
The idea that migrants are somehow better suited to farm work, or that the people of the UK are entitled to better jobs so people from other places should be brought in to take care of the crappy jobs, is the issue. I'm still not sure how this is supposed to be the "non-racist" view.
There are internal reasons that there are not people willing to do these jobs in the UK. Population figures are only one, and one that could perhaps be partly alleviated by responsible immigration, at least short term, but that isn't what is going on.
Instead, large numbers of immigrants are being used as a way to prevent addressing the other significant internal issues, and especially, to keep wages low, and avoid investment in good education and training.