Crowborough in East Sussex is very middle-class and they have asylum seekers in a former army training camp. The locals are very pissed off about it and have a demonstration every Sunday morning.
It's right on the southern edge of the town, and in the Ashdown Forest, so only close to a tiny number of homes and you wouldn't really know it was there as you drive past. I've driven through the town centre a couple of times and you certainly don't see groups of men hanging around.
There's also a huge amount of misinformation, eg that it's costing the council £500k pa, when the only info that I've been able to find is that the council got nearly £300k from the Home Office and are getting (iirc) £100 a month for every asylum seeker accommodated there. There are also stories about an increase in crime that I haven't able to substantiate, and the town is among the safest in the entire police force area (two counties and a city, so a fairly big area).
My BIL and SIL live there, albeit nowhere near the camp, and I've gone very low contact with them because they seem to have nothing to talk about other than how awful it is. Their latest cause for outrage is that people from an asylum seeker and refugee support group have been taking small groups of asylum seekers out for walks in the forest.
The forest is nearly 10 square miles, a couple of hundred asylum seekers would hardly be noticed even if they all went there at once!
My town is very middle class and I think asylum seekers would be welcome here. There has been an active refugee support group here for decades. I don't think there are any buildings that would be suitable though. Maybe when they close County Hall in a couple of years, they could use that.