There are 'vastly differing interpretations of where that point lies' - so vastly different as to render utilitarian liberalism unintelligible. It doesn't work. For one thing, it may be quite morally defensible to cause harm in certain circumstances (putting someone in prison or shooting a crazed guman). You may then claim (as utilitarians do) that it is only okay to cause harm to other people that are causing harm. But what do you mean by harm? Physical harm? Mental harm? Emotional harm? It is nigh on impossible to form intelligible value judgements on this basis alone. Is an oligarch evading tax doing anyone any direct harm? No, but he is doing is still socially unjust. Is the man seeing the prostitute necessarily doing direct harm? No, but he is seeing women as utilities to be bought and used.
Think of how you would raise a child: you would not simply say to them: 'you should just do what you want as long as you don't do harm'. You would instil them with moral values that informed the complex interactions that they would have with others throughout their lifetime.
And really all these ultra-libertarians who rail against socialists and moral conservatives alike: they fail to see the dark side of the world they have created; that the sexual revolution has led to a demotion of people from human subjects to commodified objects. Anyone who objects to their belief that subjective pleasure and self-interest should be the driving force of society is blasted as an 'authoritarian' or a 'moralist'.
And the internet: no one is allowed to suggest that those Silicon Valley hippies got it wrong; that the cyber-utopia is a cyber-dystopia. The internet has overturned the generational divisions and the distinctions between public and private that underpinned organised civic society. It has resulted in ubiquitous commercialism; the sexualisation of children and criminal networks beyond the reach of any law enforcement institution.
This is what I reckon feminists should do: rather than go on about Page 3 and song lyrics, they should campaign for ISP's to filter out all URLs containing extreme and violent pornography. That would hit the gangsters trafficking thousands of women and children and make it more difficult for children to access this corrupting filth. A few feminist women and men are campaigning for this, but mostly they don't because it would restrict people's 'free choice' - the sacref shibboleth of the age. Love the way the freedom of expression now includes the freedom for some Mafioso to peddle videos of refugees being sexually degraded for entertainment. Crazy world.
There really aren't anything per se wrong with moral values. Without them we are just animals. If you think society should repudiate all moral values, then you will not have some sort of equalitarian, egalitarian utopia; you will have 'Lord of the Flies'. Already all moral and epistemic authority has been undermined. Everything is predicated on emotion, sensation, base instincts. And what has that led to? Trump, Garage, Le Pen, Brexit - just populist demagoguery and propaganda. Everyone banging on about their right to choose and their rights.