My son and I play football in our back garden. We use a soft ball, not a proper football. Occasionally (maybe 5 or 6 times a week) the ball will go over the fence into our neighbour's garden. We don't go knocking to ask for the ball back, just wait to see if they throw it back over. If you were my neighbour would you find this situation annoying?