realistically, it's not just about managing toilet breaks, is it. By age 5 -6, so Year 1 and 2, kids are completely independent with bathroom issues. It's more that the dropped kids will be the least behaved, and will be the ones hurting someone else, teasing, running around and trying to damage the decoration or anything else, emptying the content of their glass everywhere so an adult will have to mop, trying to take all the food and needing to be stopped and leave some to the others.
At 5 -6, children generally don't really care if their parents are there or not- mine didn't. They are still too young to be embarrassed by that.
What is rude is denying any responsibility for your child and expect other parents to take care of them, assuming either that other parents stay and can offer free childcare, or that the host is Mary Poppins and can take care of 40 kids.
it's a child birthday party, not a pre-school.
At the very very least ask, but not in a way that a polite host won't dare telling you not to be rude and stay to look after your child, who needs it! It would be lovely if the dropped kid was the lovely one, polite, cheerful and a pleasure to have around. In real life, it really is not!