So all those Brexiteers and Lexiters who didn't vote on the basis of immigration controls, where are you? Will there be marches against May's interpretation of your vote?
You know, I actually believe that plenty of people didn't realise what their vote was going to unleash despite being warned frequently and emphatically. It's not too late to speak out and stand up against it. If you didn't mean it, don't hide behind that you haven't meant it. Do something about it.
Pro-tip: declaring xenophobic things to be not xenophobic, actually is not speaking out against it or distancing yourself from it. Declaring hurt at being lumped in with the xenophobes without outright condemnation of their rhetoric doesn't count as standing up to it. Much the opposite.