I have a theory on this.
I have indoor cats and I actually love seeing the neighbourhood cats in my garden.
I have made friends with them (although some newcomers are really nervous)
A few if I am out in the garden will come over and see me and get a stroke and a chat.
I don’t have any shit in the garden.
I have never witnessed any cat even doing a wee.
In the house however Dp hates it when one of our cats goes into his bedroom.
He gets really angry and shouts at them to stay out of his room.
His room has become like a battle zone. One particular cat (the alpha male) has taken to slipping into his room whenever Dp opens his door and jumping on a wardrobe so he can stare down at Dp to annoy him.
The cat has even left him presents to show I think that he has claimed a victory over Dp.
I think if you are kind to the cats they will pass by your garden and do their business else where. I think you have turned it into a battle and if they can shit in your garden and annoy you then for them it is a victory.
FWIW dcats use litter trays, they never go anywhere else apart from dps bedroom.
We have tried to convince Dp that if he was more open with the cats coming into his room they would lose interest.