Unless you think he'd harm your cat, ignore!
That's the only thing I'd be worried about. Some people are very cruel.
Your cat wouldn't annoy me by coming in. I'd just get one of those cat flaps that works on a chip and chase him away if I found him indoors. Or I might let him stay if he was no trouble and wasn't upsetting my cat.
What would annoy me was if he was shitting in the garden a lot. As long as you have a litter tray to deal with at least some of the mess, I'd be happy. I'm not a terribly keen gardener so I've never noticed a problem and quite a few neighbours have had cats over the years and now I have one.
I noticed your neighbour can't be arsed with a litter tray for his cats. That's outrageous. So everyone else has to put up with his cats but he feels free to complain about yours? I lost all sympathy for him as soon as you said that.
My cat and I are saints in this respect
. Like I said, I have a litter tray and I know for sure that he doesn't go anywhere else because he never leaves the secure garden and he's never allowed in it unless someone is in. I work from home so he can go out almost whenever he wants.
But I realise that's unusual.
He's an adopted cat and was exclusively indoor until I got him. I don't think that was cruel at all. But he likes pottering and snoozing in the garden and likes going for walks with me.
I wouldn't put him on a lead, though. He'd regard that with contempt and wriggle out of it. Cats have no shoulders so they are like Houdini.
Plus, I look mad enough taking a free-range cat for a walk as it is
.