One of my Dad's close friends babysat us when we were little. Nothing untoward ever happened to me, and as far as I am aware to my siblings BUT it came out years later that this man carried out an enormously serious sexually assault against a very vulnerable minor.
His victim was most likely chosen because they would have been unable to speak up or report it, and may not even have been aware of the crime that was committed. Perhaps my siblings and I were never at risk but it feels like quite a dodged bullet.
So no, I never would. Of course it's not all men, but it's almost exclusively men.