If you saw a group of people on the street shouting at a black person something like "Go home [N word), f off you (W word)", with the black person being visibly upset, would you seriously be thinking "Great, freedom of speech in action"?*
That would be racially-aggravated assault/abuse of a specific target, who would feel intimidated/harassed and hold reasonable fear for their personal safety. So no, I wouldn't view that as a legitimate evocation of freedom of speech.
So you'd be happy with them saying it in your house, possibly upsetting your friends and family? And you wouldn't want to have the right to tell them to shut up?
If someone in my house offered the opinion that "black people should go home", my response would depend on intent.
If there was a black person sharing the dinner table with them, I would view it as seriously as the first scenario - a deliberate attempt to abuse a specific target, who would feel intimidated/harassed. In this case, the person would be removed from my house, where they can create free assembly with other people who may feel the same.
If there was no obvious target for the opinion, I think the person's right to freely give their opinion, where it cannot be thought to cause direct harm, is acceptable. I would call them on it though. And of course, it would be the last time I had them round to dinner....
As an aside, I was once frogmarched from a room by my mother after an older gentleman expressed the "Adam and Eve, not Adam and Steve" opinion....
The above are just my thoughts, no idea how it fits in with the law re: freedom of speech, hate speech etc.