Two lovely cats, both daytime-only outdoors ranging. They're brothers, and about a year and a half old. They are our delight, and we love them and they love us.
Since we started letting them out unrestricted (I tape the cat flap open because it's really stiff - let them out from about 8 or 9 and in by 7.30 or so) their habits have changed - they absolutely love the outdoors. By the time I get them back inside in the evening, they're tuckered out. They've discovered the delights of small rodents (cats will cat!) but we live at the edge of a village in a semi-rural area, and I'm not overly worried about their predation, which doesn't seem to be happening that much.
The larger of the two (and dafter) tends to stay close to/in the garden, comes when he's called. The smaller (and definitely the hunter of the two) very determinedly goes off in the morning and we may not see him for several hours at a time. He won't always come when he's called and we've had many anxious evenings calling and calling until, eventually, he strolls up. He's practically kitten-sized compared with his chunky brother - they're different builds. The vet says this is fine and that he's perfectly healthy, but it can be hard to get him to eat at home.
Anyway, yesterday it took ages to get him in after not having seen him all day. I was seriously worried he'd met a car or been locked in a shed. Eventually, he appears, we get him in. I notice he smells of a perfume that isn't mine. (I'm fairly sure it's not something from home I can smell on him, but wouldn't discount it entirely.) Again, today, after he'd been gone a while, we get them back in and again, the perfume.
Because I suspected he might have been enjoying someone else's hospitality at length, today I bought a very lightweight collar, intending to write something on it like 'please don't feed this cat, please don't let him in your house, he is loved', if he tolerated it. He really, really didn't! 
I stupidly left the bell on and it scared him so badly it was a couple of minutes before we could catch him and remove it.
So conveying a message to a potential Lure-er via a collar isn't going to work.
I know cats just do this sometimes. But I can't bear the idea that we might lose him to someone cheeky and unprincipled. We love him so much. We pay for his vet bills, food, holiday care. We make sure he's safe.
We live on a road of 22 houses basically set on a road out of the village, surrounded by fields. Realistically, it's got to be someone in one of these houses. It can't just be a stray stroke (I'm not that petty!), as he really does smell of perfume. So - would I be unreasonable to try and nip this in the bud? Could I go door to door and explain nicely that I don't want him in anyone else's house? Could I just put a flier through the doors? I do know a handful of our neighbours obviously, and some of them are friends, so I'll start just by messaging around.
Am I being absurdly controlling?? 
I know a couple of friends who have lost beloved cats because some cheeky fecker down the road just decided 'I'll have him' and lured them away. I can't stand the idea of this happening to us.