Ignoring the inevitable leap straight to racism, the 'points based system' was always really just a vague phrase.
The actual point was that immigration, whilst not remotely a problem on a national level (net contributors etc) was causing problems at a local level (unskilled economic migrants, that is).
I saw a clip on Newsnight last night of IDS, who put into words what it is that I (and I think a lot of Brexiters) would like to see.
He basically suggested that free movement of workers continues (as it should, and as we need), but with the control & decisions resting with the UK, on our terms.
Sectors that needed to recruit outwith the UK can carry on doing so (skill gaps etc), but for unskilled work companies cannot automatically recruit externally - they would have to prove that there is a need to do so.
That was the sticking point re immigration for the majority of people, the overwhelming surge of unskilled workers in concentrated areas - usually from poorer countries of origin, therefore more susceptible to being exploited by the big employers - allowing the employers to keep the 'maximum wage' at minimum wage for most unskilled work.
Nationality, country of origin, colour of skin, religion, etc etc etc had no bearing at all on the vast majority of voters (obviously a vocal minority are the exception, but any right minded person does not subscribe to that view in my experience).