We have no garden, but where we live we have a U shape of terraced houses and my house and another links the two ends with walkways underneath. Inside is a green. Over the summer I have been letting DS play out there with next doors DS (5) - they take out a basket full of toys and muck around in the bushes etc. Another house had put a slide out there but it broke a year or so ago. When one of the owners grandchildren come (few times a week) they play outside. THe other side of my house to the green is the garages. We are off the road, then into a cul de sac then into another cul de sac. Both sets of parents (when DH is here) pop out, have windows open - I leave all my windows open, front door open, pop in and out bringing snacks/drinks/toys.
I was always allowed out like this, both boys are good (but obviously we know they are young so check on them alot, I wash up infront of the window etc.)
3 incidents have happened in the last 2 weeks though and I don't know whether I ABU to this or not.
THe other week DS took his cuddly toy and walked up the verge abit (garages side - big grass verge) and sat next to my car waiting for DH to come home. This is within sight of my living room window. Where he sat was about the corner into our cul de sac and someone who lives over the other way was loading her car, as was one of the people in my U bit (who is a childminder). She was asked by 'loading lady' if he was one of hers - she then shouted through my open front door my child was on the road (?!) and she went to get him. I went out to get him and she was bringing him back (all 10 odd steps) and I said thankyou and sorry (?!) and she told me off saying I should be outide when he is. Yes, I'd love to but I have a 2 year oldand a 10 week old and although I sit out there sometimes, DS spends whole days out there with his friend. They love it.
On Thursday DS wanted to go out there when we got in from school (he's just started so tiring) and he curled up near the front door and fell asleep and childminder ladies adult son knocked on the (open) door and told me. 
And thirdly today, DS and his friend were playing out under the archway infront of my door - there are bushes across abit of 'road' of the cul de sac and they were playing abit in there too and a neighbour came back with her 7 y o son who sometimes plays with them (but isn't allowed to pick up other childrens toys etc. because of germs) and when DS and his friend were asking him to play out, and he was asking his mum if he could she said no and said my DSs and his friends parents didn't care about them !
:( I felt like running out there but have hid inside instead - I got DS and his friend in and put on Chitty Chitty Bang Bang in the playroom, but then the 7 y o must have been allowed to associate with the rifraf out and started shouting for them, so they are all back outside now 
AIBU? I am careful with my boys, but don't want to smother them or be a helicopter parent
- when they go out I let them climb, dig, roll and just peel off their clothes and take PJs and wipes for the journey home kind of thing.
DS is back out that in old clothes, generally dirty from beind in bushes and making a construction site - am I being neglectful? Honestly? :(