Except it's not just me who thinks the cat-owners have done nothing particularly wrong.
We don't know them, their background, their age group, their experience in life, their beliefs, whether they recognise the neighbourhood children and know roughly where they live, whether they even like children, whether they have children, whether they let their children run about back alleys, etc etc etc.
If someone does me a kindness, no, I probably wouldn't invite them in, as I don't like randos in my house. But in this situation, who knows, I might. I certainly wouldn't have a red light flashing in my head, Oh, Stranger Danger! I must not let these children in.
That is for the children's parents to impress upon them, not for everyone to walk around thinking, I am a stranger! Must be alert to the fact. It's ludicrous.
These children are not mature enough to keep to OP's rules, and/or she has not impressed them upon them fully and suitably. The end.