I'd be irritated by this. My PiL have children/teenagers on both sides of them and there are balls constantly coming over at the moment. They're both nice families and the parents keep asking the kids not to kick balls over the fence but it seems to go in one ear and out the other. They've had plants knocked over, a bit of fence broken, small grandkids smacked in the face by flying footballs.
And they always want the ball back NOWNOWITHASTOBERIGHTNOW, which generally involves my FiL, who is home the most, and can't walk very fast, getting up, opening the patio doors and going out to throw them back. Only to be expected to repeat another five times in the next fifteen minutes.
They've now started ignoring the constant requests - the balls get returned the next time someone is out in the garden. FiL keeps one of the curtains pulled slightly across the patio doors, so the kids can't see him if they look over the fence.
I was once cooking dinner round there. Everyone else was doing something elsewhere in the house and I was in the middle of cooking, and next-door's teenage son was hanging over the fence, trying to get me to go and get his ball. The kitchen window was open so I said "I'm doing something right now, you'll need to wait." He quite clearly heard me, but he just stayed there, waving his arms at me, as though I hadn't noticed him, and saying "Excuuuuuse me" over and over again and pointing to the ball.
I did eventually snap at him "I said WAIT. Your ball does not take priority over our dinner." His mother came out and had a go at him and sent him round to apologise. Funny, he seems to have managed to keep the ball in his own garden since then. The other side aren't doing so well, though.