Wrongtrouser is right.
There is nothing intrinsically "left wing" about freedom of movement. In fact, it was next to impossible to move from one city to another in the USSR without being assigned a job by the government, let alone move to another "country" in the republic without official sanction.
Indeed, back in the 70s and 80s, immigration was seen as a tool of the capitalist to drive down workers' wages, smash trade union power, or to uphold profit margins in a failing industrial climate.
This is why you got some old style left-wingers supporting "Labour Leave" and some firebrand trade unionists arguing against the EU. I remember left-wing arguments in the early 90s about the EU and freedom of capital and labour and how it would demolish worker's rights and the ability of governments to financially provide welfare and support to workers because they would not be able to tax capital across borders -- an argument that proved prescient with multinationals tax-hopping through the EU.
To me, what is being perceived as a "right wing upsurge" is actually a phenomenon where those who would traditionally have expected the "left" to represent their interests (in Britain, this would be "old" or "traditional" Labour voters) have thrown their lot in with those on the protectionist right because they share a common concern about the consequences upon their lives of the excesses of global capitalism.
In my eyes, this is not "right wing upsurge", but more an "anti-globalism upsurge".
You can see this clearly when you look at how the phenomenon is manifesting with Brexit and Trump. Trump basically won on a pro-American message of jobs, industry and the economy through policies that would control capital flows and the products of capital (anti-TIPP, anti-NAFTA, imposing tariffs, wanting to design laws so that the process of production and subsequent profit has to be repatriated back to the US etc). And control of capital to favour the worker historically sits at the heart of traditional left wing thinking.
I think, with what is happening, you need to separate the economic spectrum from the cultural spectrum and also consider the libertarian to authoritarian axis for both those spectra as well.
I suspect the shock has been because people in positions of influence have got too used to perceiving politics through a cultural lens, rather than an economic one. So they are reacting to Brexit and Trump as though they are solely cultural reactions to the status quo, rather than economic responses: hence all the racist, xenophobe, fruitloop, racist labels, which are all firmly in the domain of the "culture wars".
"Culture politics" seems to be where a hellova lot of people think the battle is situated, and that is why so many people are missing the point and why the situation is getting so explosive. It isn't about culture, it's about the economy.
With big shifts like this, it is always about the economy.