This is why:
www.walesonline.co.uk/news/wales-news/wales-most-pro-brexit-town-15767233
Money not spent in productive, long term projects.
Vanity projects, that are foisted on an area.
Little or no communication with the local community
Look at those unemployment figures, and try telling those people how good the EU has been for them. As a PP said, it's primarly the middle classes who benefit the most from travel, plentiful employees etc.
And yes, people ARE worried about immigration, particularly when there are few jobs to be had. A sudden influx of people is hard to absorb, especially in smaller, more homogeneous communities. And (unpleasant though it may be for some to admit) they are not all hard working, solid EU citizens. I have a relative who lives in a small Valley town. There has been a large increase in Eastern Europeans housed there - shipped out by Councils as rent is cheaper . They don't seem to have found much proper work and in my relative's words, spend their time "Fighting, drinking and pissing in the street".
And no, saying that does not make him a racist. I wouldn't like to see that happen in my area, - would you? But then, like so many of the "shocked" Remainers, I live in a very "naice" South East village, where the ladies-who-lunch like having all the cheap cleaners/plumbers etc.
The Eu experience is not universal.