Sadly, blackcurrants I think the point you are making is:
men who want to work with children [expect] to be subject to general social distrust and scrutiny
and I do think that is genderphobic, so the accusation wasn't lightly tossed. It was pinned to you.
I've been pursued down the street by a gang of young ethnic men; yes, for a while I was cautious. Do I distrust men of that ethnicity? Nope. Just silly young men.
Women and children who have been raped and/or abused by predatory men have every reason to be phobic - that is, to fear them, in the abstract at least, if not in person. It's not fair to the individual men who are perfectly unthreatening, but it is an appropriate response to the levels of violence and harassment and abuse that women and children face at the hands of men.
Perhaps - and only perhaps - for those women and children; but certainly not for all, particularly not those who don't share either your life experience of worldview.
Men who don't like that fact do, I'm afraid, have to lump it.
No, they don't. They can call you out for it, and watch for the increase of men being driven out of professions...
Good men who observe that fact tend to respond that it is a problem with our patriarchal society
No, men who agree with your perspective might say that; that they don't share your perspective does not make them bad men, just not ones who share your view.
and how that damages women and children, not something that people should just shut up about because it makes them feel sad.
where does anyone say anything like "people should just shut up about because it makes them feel sad"
Saying the sort of things you do, will at least make the nod-and-wink "all men are paedos" conclusion easier to jump to; and it is wrong.