In discussions like these it's worth remembering that often, the answers the law settles on aren't perfect - there is no perfect answer.
What there should be is an attempt to balance interests.
You can make a good argument that private business owners should be able to rent premises or do work, or not do those things, for anyone they choose. And for the most part that's true, it's part of their free choice.
Where we restrict that in regards to protected characteristics, we are restricting their use of their own property and skills, absolutely, the state is doing that. The justification is that allowing certain kinds of discrimination is so socially harmful that it is worth the state stepping in.
It's not even just about harm to individuals. If I won't hire my hall out to someone because of his bad fashion sense, that might be just as bad for him as me refusing some other guy because he is black, and might reflect me being an asshat in both cases. But the social harms of the latter are wider and more impactful (arguably, for now.)
On the other hand, that kind of law potentially impacts some groups freedom of belief, freedom of speech, or mission in terms of intellectual or religious or political activism. So the law allows certain kind of exceptions, because the larger harms to democracy would be severe.
There could be other ways to balance it. The loss and gains on all sides are not easy to weigh. And needs might change over time, what was working 100 years ago might not work now.
I do think where we run into trouble can be where protected characteristics are brought in that are around really socially controversial ideas. Gender identity is interesting in that light because no one really seems to know what the heck it is supposed to mean very clearly. (So in a certain sense, they only work smoothly where they are largely socially accepted already. )
And the more of protected categories there are, the more we are restricting what people can do in line with their belief systems, which is a good argument for being very conservative about defining them.