Don't you think as Leaders, we don't have the ability to sniff out a dodgy bloke in a dress & say no to their help?
I agree with most of what you say including the value of male volunteers. (DH used to help out with the occasional nursery school outing when they were little - he enjoyed seeing DC with their friends, he gained respect of the staff by helping clear up vomit, and presumably it does no harm for children to see a normal male in their daytime environment. A message that all men are dangeerous is pretty damaging.)
But as a volunteer in a youth sports club, and after a recent conversation with a boarding school matron, I would say it happens more often than you think. Because of data protection and more, schools, groups and clubs wont tell you why someone leaves. My spidey senses worked once, which led to me receiving real abuse from other parents who did not understand why a popular coach suddenly left the club. However no one, but no one, was prepared when a club member was arrested and convicted for offenses that happened outside the club. Clubs, schools, extended families, friends and neighbours. Trust is needed, but so too is common sense and vigilence.
Which is why Stonewall et al should not be used a experts on child protection. Yes they may have some useful insights into how to be supportive of adolescents who are struggling with sexual orientation, but gender is different. Youth organisations need to be able to apply segregation by sex to ensure safety. Stonewall and friends are lobbying for segregation by gender. Not the same....and dangerous.