The cost of those rights is having to deal with the fact that fuckwits will also have the right to speak, so stuff like Little Britain will still exist, but I think it's a cost worth paying.
But you are not paying that cost (assuming you are white, apologies if I'm wrong)
In essence, we are saying that any minority group has to put up with an enormous amount of racist speech and culture that white people simply don't have to deal with.
I don't think it's as straightforward as saying free expression is the best protection for minority groups. Free expression also directly leads to physical and mental harm being inflicted upon minority populations.
Look at Trump's America -- concentration camps for migrant children, an enormous spike in hate crimes and white nationalist terrorism. Free expression has not stopped any of this.
In Germany, it is forbidden for people to share Nazi propaganda or display symbols. Is this a bad thing? After all, it's censorship!
And yet, support for German neo-Nazi and far-right political parties is among the lowest in Europe. That is not a coincidence.
I don't see a problem with a country sifting through its culture and having a conversation about whether some things should be put away or put in context. Times change and we don't have to celebrate the same things forever.