I get where you're coming from. But...
Imagine we're living how our ancestors lived for aeons, in a small settlement. Protective walls, lookouts, a few houses, livestock etc. You see some strangers coming over the hill. They're not from your tribe. They might be bringing interesting new goods and knowledge, or they might be coming to steal from you, spy and wage war. Do you let them in the village?
There's no right answer, but I think the basic principle of whether you see outsiders as an opportunity to learn and grow or a threat to be guarded against is at the core of being left or right wing. Are you defensive or expansive? There are risks in both.
I don't see ethnic minorities as not belonging or being outsiders and I don't think there's anything to fear from them. But if you live all your life in a small, defensive way then you think your community is at threat from them and you want to defend it. You're afraid of change because it might hurt you.
The same with economics - I have never and will never vote Tory but I think it's a valid argument to say that subsidising everything isn't the right way to do things and incentivising personal initiative and innovation is a better way to go. That's what's at the core of the different between left and right, in theory. In practice it's all a corrupt pile of sludge.
So basically, the kind of narrow minded racist voters you're talking about (have some in my family too) I feel hostile towards but also sad that they live in such a small world. Tory voters per se I think can be honourable, decent people who just have a different view of the best way forward than I have have.