We've got this "problem" too. The complication is that our male Siamese is extremely sociable and likes to go visiting. Not a problem in itself, but......
There are some days we don't see him all day. Especially in sunny weather, there are various places he likes to go and sleep/chill out. One of those, for example, is in the long grass underneath the raspberry canes. 
One place he goes "visiting" is the neighbours at the back of the people across the road (ie they're in the next street), who then feed him. They have other cats but don't seem to be used to the character of orientals.
This morning we got rung up at 7am and said Cosmo was over there and didn't look well. Last time dh had seen him was at 6.10 when he'd got up and disturbed Cosmo who was sleeping on our bed (as he does most nights) beside our other (female which means she doesn't wander ) Siamese and had looked fine. Dh went rushing round and there was Cosmo sleeping in a wee hutch that they have. They said that doesn't seem normal behaviour, so they'd given him some food. 
Dh, as politely as he could as this isn't the first time this has happened said, "He's a cat , he's enjoying the dry weather - and please don't feed him" and called " Cosmo!! ". The cat jumped up and went across to dh, purring, so dh then walked him home (he's a dog-cat - will walk along beside you).
We've now shut the cat clap and are keeping him in for the morning much to Cosmo's extreme and very vocal disgust
Don't know how many times we've told these neighbours that he is a sociable cat, he likes to go visiting and of course if they keep feeding him he'll keep on visiting
I asked dh if he'd told them today that Cosmo has dried food permanently in his dish but he said he hadn't as he was trying too hard to stay polite and to just remind them again not to feed him.