It's foolish wrong to blame "them foreigners" for everything that is bad with the country, BUT it is equally foolish and wrong to immediately dismiss anyone who is concerned about immigration as a racist (which I'm not saying the OP is doing, by the way).
What happened to the OP sounds offensive and shouldn't happen. But rather than just say the people who did it were just a bunch of racists, bigots, Nazis or whatever, will not do anything to change the views of those people. Changing their minds is surely what we want?
Throughout history people have blamed "them foreigners" for whatever problems their country faced at the time. It is right to challenge people on these views, but it is wrong to deny that whatever problems people blame on foreigners do not actually exist. The problems do exist, it's just that perhaps people are laying the blame in the wrong place.
Find out why people are blaming immigrants. Why do they think that they are taking our jobs/benefits/whatever? Above all, show them - with real, practical examples - why these people are actually better off because of the relaxed attitude to immigration. Explain to them why we are better off as a country and why they are better of as individuals because of the (admittedly huge) number of immigrants we've had over the years.
People who want to "kick 'em out" or "send 'em back" will not be persuaded by ideals like "free movement" or anything like that - unlike you, they do not want to live in a world where anyone can live anywhere (of course, that's not the world we live in either). They need to be persuaded by tangible benefits of immigration. Things that they can actually see and feel - especially the roof above their head, the job they work in, the money in their pocket. Why are these things better for them with immigration?
If you can come up with a reasonable argument to answer this last question, you will be able to persuade an awful lot of people. There are racists who will always be racist, but there are many more people who are not anti-immigration on principle - they just want it to be demonstrably better for them personally.
If we can't answer this question and show the likes of the people who started on the OP why they are better off because of immigration, we really can't begrudge them their opinion. And certainly we can't expect to change it.