I blame TV makeover shows. Gardens are not extensions of your home. They are colonised outdoor areas. You can control their environment only to a limited extent. Accepting that wild animals will roam through them (and shit in them) is as necessary as accepting that you can't control the weather in the garden either. Anyone expecting a sterile garden environment is foolish. If you are worried about immuno suppression or children touching things, no one should be allowed out into a garden without it first being checked for all sorts of wildlife droppings (and cats). Most wildlife droppings will contain far more hazardous materials than a well-cared-for cat's and in the case of birds and rats not necessarily be very noticeable and therefore actually much more risky!
Most cats bury their faeces, so if it's on the surface, it may well not be cat faeces anyway. Cats are also territorial so tend not to defecate too far from home. In a cat dense area they will normally limit to their own garden. Cats pooing in other people's gardens isn't as much of an issue as people think and the culprit is often another, wild animal.
I don't think it's 'ok' for cats to defecate in someone else's garden, but at the same time I think it's often a misdiagnosed problem when other animal's are responsible, and in comparison to other wild animal's defecation getting upset about it seems a rather pointless exercise. If you removed all cats from the face of the earth, your garden would still be horribly 'contaminated' by animal faeces.
Cats are domesticated animals but they are not human dependent in the same way as dogs and their natural behaviour includes outdoor exploration. Just as it's cruel to keep a chicken in a battery cage, it's generally cruel to keep a cat indoors (there are exceptions where people have set up their home to include grasses and plenty of clawing/exercise/mental stimulation opportunities). This is recognised in UK law, hence cats are allowed to be free-roaming. Short of putting cats in nappies, this inevitably means free defecation also. It's not so much that cat owners don't care about other people's gardens and more that they recognise the species' needs and that it is legal. Cats are effectively in the same legal bracket as wildlife.
Disclaimer: I am a cat owner, my cats don't go outdoors (they come in to use the litter tray and then go out again!), and I keep them in from dusk til dawn to help protect birds. I also have a dog and recognise the different species needs and legal status so he is microchipped, kept on a lead apart from when I can ensure safe, free-running opportunity, regularly deflead and wormed (as are my cats), insured and I pick up his faeces. Different species, different needs/treatment.