Surely the answer to this is to have some sites that allow small children, and some that only allow older kids or adults (as well as adult only sites) and that way people can choose to risk whether they will be with babies and toddlers!
The last campsite I was on, virtually everyone on the site had under 5's, with a lot of tiny babies, toddlers etc. When they cried at night, they got sympathetic looks the next morning, and as parents we were generally apologetic to each other.
The first night, my 3yo screamed inconsolable for about 2 hours at about midnight. There was nothing I could do (she has health issues which means she has inconsolable tantrums occasionally). Obviously we didn't go home, because driving floor 2 hours in the middle of the night with a screaming octopus isn't sensible, and would either mean abandoning the rest of the family in a field without a car or way home, or my husband single handed packing up at that time in night, whilst I comforted said screaming octopus, and our 4yo chucked out of her bed.
Obviously we appologised to people the next morning. They were lovely and days it would probably be their turn soon. And they were right. Or little one didnt cry like thst again, but others did.
Part of being on a family campsite is that there will be noise from children, and crying etc.