trampoline dilemma - do you let all the neighbours children use yours?

[shrub]

we bought a trampoline for our ds's (5 and 2 years) which they love. the problem at the beginning was our next door neighbours daughter wanting to use it, not a friend of ds and was also expecting my 3rd baby so explained not at the moment as i would find it difficult supervising other peoples children/health and safety etc. she said no problem - she would supervise! (don't really want other people in my gardenstill said no but gradually over the holidays felt like the selfish giant and in order to be more sociable having only lived here a year relented and now have all the children from the surrounding neighbourhood all wanting to go on it - not only get the neighbours children but their friends aswell who aren't even friends of ds. they don't even say 'can we play with *?' they just come out with 'can we play on the trampoline?' feel i should have been more assertive from the beginning as find it difficult to say no at the best of times especially to children. when i say not at the moment etc. they then just stare over the wall and watch us using it!
anyone had this problem?

Our neighbours have a trampoline for their six year old. They have removed the fence panels between them and their neighbours, as have the neighbours. Basically, all three houses have access to the trampoline and other play equipment.

Unfortunately, the trampoline is right next to our fence, which means gaggles of children making enormous amounts of noise next to our house and jumping up to watch what we are doing in the kitchen or on the patio.

It's very kind of them to invite the whole street to play, but when they are out, or on holiday, these children do not respect their property. Last time I looked, their neighbour's son was throwing decorative pebbles at their conservatory and the girls from the next house along were emptying the contents of the playhouse onto the lawn during a rainstorm.