Yeah, this is a much more likely scenario, IMO/IME. My cousin has just got one child, a daughter, roughly the same age as my two. (all now in their 20s.)
She used to live in a street with 4 families with 3 and 4 children, all within 4 or 5 houses away. She would have a little trampoline and mini bouncy castle and a little pool in the big side garden (end council property - so big side garden,) for her little girl when she was small (under 7.) Some days she had them all up, not all.
Every single time, within 5 to 10 minutes anywhere between 3 and 6 of the kids in the street, would let themselves into the garden to 'play' with little Lucy, even though she was only 2 or 3, and they were 6 to 9 y.o.
The parents NEVER asked, and never came across to watch them as well, and sometimes fucked off out and didn't come back for 3 or 4 hours and just left the kids with her. She was feeding them, looking after them, making sure they were okay etc.
Sometimes they would just turn up, and sometimes the parents would call over and say 'they want to come and play with little Lucy. So I'm sending them over.' And then they'd just go off out for like, half a day and just leave them there. Sometimes didn't come back til 9-10pm. Used my cousin as a fucking childminder!
I find parents of 3 or more children, a lot more cheeky, and a lot more likely to dump their kids on other parents than parents of an only child are to dump their one child on someone. I've never known that happen actually. (In fact parents of 2 don't do it much, it's the ones with 3+ in my experience.) Had it happen to me a few times. Happened to my cousin a LOT though, as she lived in a street with families with 3 or 4 children really closeby. There has always been very few children closeby wherever me and DH have ever lived.
tl;dr YANBU @Theseers But I am not sure what you can do about it.
nb; not ALL families with 3 or more children are cheeky feckers who dump their children on others, but some are, and moreso than parents of 1 or 2 children IME.