Okay, I will give an example of how being in the EU has affected people where I live.
Local authorities have to plan for house building to meet local demand. Because of being in the EU and high immigration numbers, the calculated projected figures for the increase in population are very large in this area. Because of this, the local plan for this area (pretty rural, smallish market towns, small villages etc), which is currently being drawn up includes huge swathes of new houses, often in huge estates and often next to fairly small rural villages. Local councils cannot just decide to ignore the projected figures, or set lower house building targets or they will be penalised in various ways by the government. In any case, if you have a large increase in population, you need more houses so that would not solve the need for housing.
So membership of the EU leads, in this area, to a large increase in population which leads to high numbers of new houses which many people consider to be changing their area, very quickly, in a way they do not want.
Someone will no doubt argue that this is the government's fault or the local authorities, but if there was not a large increase in population there would not need to be huge numbers of new houses built.
So this is a concern, which I'm sure is held by many where I live, which is to do with immigration, but I really can't see is racist (please explain how, if it is) and is to do with EU membership.