Yep.
Freedom of speech doesn't mean supporting just the speech you agree with.
It's an unpleasant fight to have, or thing to defend because really often you are forced to defend stuff you find stupid, horrible, offensive or wrong.
But the principle is so important that we have to keep doing so.
The very instant we start to think that speech we disagree with should be oppressed on the basis we don't like it, we are undermining our own position and our own freedoms.
I suppose in some ways it's got a similar discomforting factor as safeguarding, because that also goes against some deeply ingrained behaviour.
In safeguarding, we assume the worst of everyone, and keep doing so. This isn't easy for humans to do. We naturally want to give people the benefit of the doubt, and trust certain people. It's mentally taxing to keep suspecting people, running through worst case scenarios, etc. But it's essential to maintain the underlying principle, which is to guard against laxity in safeguarding/loopholes/sacred castes, etc.
When it comes to freedom of speech, I think we have to be doubly careful when looking at groups whose speech we disagree with, because the natural inclination is to say 'shut them up'.
And then of course there's the risk of over correcting.
All of this is complex and not easy, but it's all essential to a functioning democracy, and none of it ever ends. There is never a point you can become complacent about freedom of speech, or safeguarding, it's all a constant, ongoing, ever evolving process.