Ok, here's my take.
Anyone I leave my kids with, I will assess, consciously or not, whether that person is likely to care for my child safely. Everything from 'what are the chances of this person being a paedophile' to 'is this person too absent minded to be safe caring for my child.'
For myself, I'm probably more likely to assess things in situational terms, consciously or unconsciously. And depending on the situation that will involve considering the likelihood that any men in that situation may assault me sexually, because that is not uncommon, and because I've been socialised to do that.
If we're being semantic about it - as this thread seems so keen to do - from an individual's perspective any man could turn out to be a rapist, and any person, male or female, could turn out to be a paedophile. One is between twice to four times more likely than the other depending on what stats you read, which will of course affect people's risk perceptions. However having been at a school where a number of children were abused, there is a risk of over-minimising the likelihood of paedophilia.
My point in saying that assuming make child care workers were in that career because they're paedophiles is a misandrist statement stands true - because there is no evidence that men who choose caring professions are more likely to be paedophiles or rapists.