So, this is interesting.
My son has severe intellectual disability, autism and is non verbal. He is 11. During the heatwave he was in our paddling pool naked, the happiest he has been in a very long time. Utter joy! He'd get out and run round the garden then get back in. If anyone knocked on my door or sent me a letter etc, I'd be telling them to sod off.
Firstly, I'd suggest that they educate their own grandchildren that they shouldn't be staring into someone else's garden, not least via a trampoline. Nosey or what?! Can you imagine if someone sat and stared through a hole in the fence at you/them. It's not dissimilar.
Secondly, it was in our garden is because it's OUR space. I certainly wouldn't take him to a public place and let him do that! We have a wall and fence and as far as I'm concerned, as long as its not out in the street it is literally no one's business.
Even the most innocent of children can't enjoy simple pleasures because some busybody takes offence by looking at something (aka a garden) that's not theirs.
Their comments should well and truly have been nipped in the bud.