I think it is not possible.
One of my favourite quotes: "freedom of speech does not mean your actions do not have consequences."
If Boris had said he doesn't agree with the Niqab fine that is freedom of speech. But to use terms he did just was harsh and uncalled for.
Unfortunately many right wing believe they can say what they want. Tommy Robinson is a good example he nearly ruined a court trial because of his illegal reporting.
You need to protect minorities whether they be: women, ethnic minorities, gays, Muslims, Jews etc...
Is it right to say a women at work "go make me a coffee" "is it your time of month" of course it isn't. This is where freedom of speech must be limited.
But unfortunately many who allow hateful speech and say "it's my freedom of speech" tend to have racist views.
"it's not racist to not want refugees"
"immigrants should go home"
"I am not racist I have a black boyfriend" 
The excuses people make you get shocked
It is 2018 and still people need to say lines like "I'm not racist but" (something racist comes out)
The black friend, boyfriend one is the worst though