I think where you live in the UK has an influence on how you see immigration, whether it's EU or Non-EU.
For a lot of people it is seen in terms of economics. It's a fact immigration (of both sorts) brings in much more to the economy than it pays out in benefits. It is also a fact that immigrants do really important work in the NHS, Care Sector and various other sectors and services.
However, I think that is where some people end their opinion, which is more than fair, but I do think there are further things to think about which may have caused other people to fester concerns.
For example, the claim of "they are taking our jobs" is wrong and a bit silly. But in a very small number of areas (usually poorer) a few firms have recruited by only advertising and interviewing people from the EU, this has been confirmed in the News etc. Linked to this example is the claim that "immigrants are driving down our wages", this is also not wide scale but it does happen. There are a few companies that have hired immigrants who don't know the minimum wage and are just grateful for a job, this is ilegal but has to be reported first. Paying cash in hand is a way to keep it off the books and again this has been confirmed, but there are only a small amount of cases.
Even though these are isolated cases the rumours can soon be exaggerated and get around all of a town quite quickly.
Both of those examples can be solved by the UK, it's not the immigrants fault and it's not the EU's fault. Those are Council, Government, HMRC and Police matters. The trouble is if nobody explains that to the public some of them will wrongly blame immigration.
There is an issue with some immigrants undercutting in jobs like building and plumbing. Now this happens in every country in the world, some more than others but there is not a lot you can do about it. Whilst I have sympathy with those people who struggle financially because of that, but it's still wrong to blame all immigration because of it, competition is the problem, it doesn't matter where they come from.
The final issue I feel is possibly the biggest problem. This one definitely depends on where you live. Most immigrants work and live their lives mixed in with the local community. However not all immigrants and not all UK residents work. There are parts of the UK where a lot of a street will have immigrants in a row, so it feels like a sudden influx. Some of the immigrants aren't working and tend to be the poorest ones, they are then housed next to UK residents who are umemployed, poor and in higher crime areas. You will also have examples of 10 or more people living in a 2 bedroom house, the landlords turn a blind eye as long as they get more money.
If you add to that housing situation the fact that we are already overcrowed in housing terms in the UK then you can see why tensions begin to develop. The UK is massively behind on Social and Low cost homes, we aren't building enough for normal population growth, let alone EU and Non-EU immigration growth. I don't think all parts of the UK realise what is like in some of these areas.
Again this is not the immigrants fault, it's successive Governments and Local Councils. The Government has not built enough homes for decades and to keep up with immigration they need even more. Local Councils do no real planning on where they house immigrants and have no local support networks set up for them. Putting people in areas of unemployment, crime and racist attitudes is a disaster waiting to happen, but avoidable!
If successive Governments had planned better, talked in detail with the public and councils then a lot of this misplaced resentment would have been avioded and even less racism left to fester.