I didn't leave Ds1 at a party by himself until he was 5; but until then I knew all the parents of the party-children as well through playgroup. Since then, I've checked with both him and the party hosts whether or not it's ok for me to stay - seems where I live in small town Australia it's quite normal for parents to stay, or not, as they choose, so no one frets. I've taken Ds2 with me when the birthday child has smaller siblings too; but have also checked with the hosts whenever possible that it's ok for me to do that.
I have had 2 parties for DS1 that have been pool-parties and on both invitations parents were asked/told that they really needed to stay as we could not take on the responsibility for their children in the pool (big parties, 20+ children in both cases). In both cases, each child had at least one parent, if not both, stay. We catered accordingly. I did, however, also say that I couldn't allow children over 7 to come because it would have changed the pool dynamic quite dramatically - in the end I did have 2 bigger girls there, but luckily they were fairly well behaved and they didn't get a party bag as they weren't meant to be there.
DS1's last party was number-restricted and not a pool party; and still about 1/3-1/2 of the parents stayed, and that was ok. A couple of junior siblings came too and that was also ok.
In your case, OP, I would have done what you did - and would have taken Ds outside if it was a problem, but not gone too far away in case DD got upset. Glad it all worked out for you in the end and I hope the party hostess remembers to put a contact number on next time!