About this xenophobia business and international freedom of movement: it's true anecdotes are only worth so much. However it's also the case that the evidence from polls and studies suggests that attitudes to outsiders are actually quite positive compared to other European nations. Do people really imagine that the Dutch or the French don't have significant elements of race-based nationalism found in their culture, just to mention two who are sometimes considered to be extremely civilised - I would suggest that kind of thinking is far deeper seated in some other European countries than it is in the UK. I'm sure not everyone would agree but it's not a crazy idea. And Germany has had serious and significant problems with racism - directly as a result of movement of population.
And I think this is the rub - wherever we see large-scale movements of people, especially into smaller communities, and especially when they are at the lower end of the economic scale, there are upsurges of various sorts of xenophobia, as well as articulations of concerns about newcomers that aren't xenophobic or racist (and yes these are different.)
So there is a real problem for people who want very open borders along with acceptance of different sorts of people. Or who would see having both as the measure of success. There is a good chance that both would not happen even without Brexit.
The rather bizarre claim that being leftist means wanting very open borders is a little hard to take though, and so it the kind of moral superiority that seems to come with it. Open borders has never been mainly a leftist position, it is a liberal position, and is the sibling of open borders for capital. Both are always most advantageous for the middle classes and elites, and as such are at odds with traditional leftist thinking which is about protecting workers, industry and the working classes by managing immigration and movement of capital. To put it another way, the EU is a globalist project and freedom to travel is part of that. It's good for employers who want wage competition and to avoid the need to invest in training.
This mix up between leftist economic values and liberal (that is right wing for those who might be confused) economic values accounts for a lot of the confusion, and maybe anger, middle-class, university educated, urban Remainers feel towards Brexit voters. Liberal values have always been very good for them, and by imagining they are leftists it allows them to feel morally contented as well. And I think the belief that it is moral is genuine enough, but it really does impede understanding when the assumption is that liberalism and leftism are the same and both are equally good for workers and outlying regions.