These aren't my personal opinions but I do come from a small, Northern town so can speak about the local generalised perspective to an extent.
I think that because EE are seen as economic migrants and are causing people to lose jobs, that that is part of the reason why there can be some generalised negativity.
I have family members who worked in a local factory for a massive, international brand. They had a record year for profits and the same year they laid off half of their skilled technicians, a lot of whom had worked there for decades, and replaced them with much cheaper agency staff, vast majority of which were EE.
It's not really the fault of the EE workers, moreso the company for being greedy but sadly it's the EEs who get blamed.
I also think that no matter where anyone moves, there's always going to be a preference, even if unconscious, in people to their "own" people. I lived in Australia, you probably couldn't get a country much more similar to the UK, but I was still always "the English girl".