The answer to why not is "because I said so". Stop negotiating and justifying to a 4yr old about how you want to live your life. Tell him firmly that you've already said no once, to stop asking to come over after he's been told no because it's rude.
If that doesn't stop him (and it might not) tell him that if he asks again, when he's already been told no several times, he won't be allowed to come over for the rest of the week/month. That gives him negative consequences for his nuisance behaviour. For that to work, you'll have to let him over/yours go over there occasionally. The threat of "taking away" something he never has anyway, won't work.
He's not being brought up to respect others boundaries (even if they're in the house I imagine his parents can hear him shouting - and they're not coming to see what's up), that doesn't mean you can't teach him to respect yours. It takes firmness and consistency, over a period of time. You're training him, essentially, possibly to do something he doesn't have to do with anyone else in his life at the moment - respect them. Don't expect him to learn the lesson overnight if everyone else around him is teaching him something different - that he can be a nuisance without consequences.
If he still keeps hassling you, tell him firmly again to stop asking. This time don't be friendly, be cross. You've already told him nicely, several times, now you're stepping it up. Then ignore, but don't go inside, get on with whatever you're doing. Otherwise you're teaching him that if you're in the garden, you're fair game.
Your DC don't have to ignore him, they can chat through the fence.
You can insist your own DC don't repeatedly ask you to have him over/go over his if you've already told them no that day. Get cross with them if they do keep repeatedly asking. Lay down what the consequences will be and follow through if they don't stop. Your own DC need to respect your decision too. In time, they'll learn to only ask once.
If he keeps hassling them to ask you (which you've already made clear to them will get them in trouble) they'll get fed up of him and start ignoring him anyway. Which is as it should be and different from the scenario proposed in your OP. It's not right to get them to ignore him so you can avoid having to be assertive. You do want them to learn to ignore someone who's trying to get them to do something that'll land them in trouble. If he's not getting a reaction from anyone on your side of the fence he should stop. He wants attention, if he's not getting it there's nothing in it for him. When someone is starved of attention, any attention will do even if it's negative attention eg a telling off.
If none of that works either, you'll have to go speak to his parents and ask them to keep a closer eye on him. If they're decent people they'll be mortified their DC is being a nuisance and will monitor him better and tell him off themselves if necessary. If they're arseholes who let him run wild and CBA to engage with him, nothing will change and I can only suggest earphones and a podcast/the radio when you go to do something in the garden, plus the installation of a 6 foot solid fence.