Be cold and distant. Wear earphones, and if they try and catch your attention make a big thing about having to stop and remove them from your ears – make it clear they're interrupting you.
If that doesn't work, be slightly rude ("sorry...just want to finish this book while I've got some time," then carry on reading and don't look up).
If that doesn't work, really lay it on the line. "Could you stop interrupting us every time when come into the garden please. I'm happy to chat now and then, but we want some privacy"
The selfishness is staggering. They're just using you for a bit of entertainment. I've had so many neighbours like this. One used to appear literally every time I went into the garden. I'd see his head bobbing up and down as he tried to see if we were outside, then you'd hear the door. He was one of those unbearable people who think they're funny when they're not and keep making stupid little jokes. I was reading in the garden once and he came outside and just stood there with a stupid big grin as if to say "lucky you, I'm here now." I was so angry I just ignored him. He carried on standing there, and then said "what are you reading" I said "a book," and didn't look up. Thankfully he got the message and f-d off indoors, no doubt thinking I was out of order.
The guy opposite is now retired and bored. He's constantly walking his dog, and never has it on a lead. Naturally the dog (who I am pleased to see, unlike most of my human neighbours!) runs up people's drives. He's then got an excuse to pursue it and talk to you. (We had another neighbour who used precisely the same trick.) This man is slow, thick and boring. His only interest in life is cars (which I have minus zero interest in), and he just talks at me in a slow, boring voice about Formula One, or who he had a go at in a shop (he's one of those men who enjoys bullying and yelling at shop assistants and nurses because he knows they can't retaliate) for as long as he wants. I go out of my way to be as rude as possible to him now. I'm sick of it. He couldn't care less if I'm in a rush. So long as he can amuse himself for ten minutes, that's all that matters.