OP I really sympathise, I'd be exactly the same.
We have a kid down the road where we live, who turns up here pretty much every day.
He's friends with DS1 (9) at school, and gets on OK with DS2 (6), so often the kids want him to come and play. This makes it harder for me to say no.
Other times they don't want him to come in, so I tell him that we're busy. He then asks when we'll be free, and spends the rest of the day prowling around outside, knokcing on other people's doors, walking past our house, peering in through the window, knocking on the door and asking if we're free yet. We've even tried ignoring the knocking and pretending we're out, but he just keeps on knocking and looking through the window.
He's a perfectly nice kid, but the dynamic of 3 kids is totally different to 2, and often requires my input breaking up fights etc.
Each time I get to the point of thinking enough is enough and speaking to his parents, he disappears for a couple of days, and then I think I'm being unreasonable and making a silly fuss.
But honestly, it's hard and relentless, and it wears you down. It's like having a dog constantly coming up with a lead in its mouth begging for a walk - of course I have "ultimate power" but having to wield that power grinds me down and removes enjoyment from the day.
I'll admit I'm a soft touch, especially when it comes to children, and maybe it's my fault for not toughening up. But I think it's unfair that I should have to change my personality because some parents are too lazy to consider the impact their kids may be having on other people.