There was mass immigration - and a bigger working class than now - in the 1950s and 1960s though. Again, the fascist vote was not the popular vote.
Why did we have that wave of mass immigration? Because British workers were not properly skilled for prestige jobs but did not want to do low-status jobs.
Ringing any bells?
FWIW in the reign of Elizabeth I the population was more like 5 million. Overcrowding was again a worry. Yet since then we've added 50 million through one method or another, and all that without a history of popular fascism or anti-immigration on anything except the most marginal of levels.
Of course there is a story behind the story. Of course people are voting BNP for a reason, many of whom would faint at the idea that they were racist or fascist.
But let's not simplify it into, "oh, they've got no work, how would you like it?"
How would I like it? I would shovel shit into a hole for minimum wage before I would blame another man or woman's skin colour for my inability to support myself and my family.