I totally get it. Although most men aren't going to bother your son, there are a lot more sexual predators than people realize. Just look at the sex offender registry where you live, if you live in an area that does that, and you'll be shocked and appalled by the numbers. Yes, stories that are especially bad make the news and maybe people think our fears conflated because of that, but there are MANY, MANY more events that haven't. By far most have not. I was molested as a child by a number of people and so were most of the people I know, women AND men. I'm not kidding. I've known more who have been molested than haven't. As an aside, these are all very well-educated middle-class to upper middle-class people. It isn't relegated to "bad" neighborhoods or areas. Not at all. Don't assume that because you're somewhere nice or live in a "nice" neighborhood that you're somehow safe from these things. It crosses all socioeconomic boundaries.
On the other hand, your child needs to be able to be independent and he is getting to the age were it is uncomfortable and embarrassing for him to be in the women's room. Try and find family restrooms when possible. When it's not, wait right outside and listen for any improprieties. He should only be gone for a few minutes at most and you ought to be able to hear if anything bad is starting to happen.
From what you're saying, you are reluctant to tell him WHY you are concerned, but he should be aware for his own safety. You don't have to go into graphic detail, but explaining that some men will try and touch children. Tell him to not talk to strangers, ESPECIALLY men who are being overly friendly when you are not present to be able to suss the person out yourself. It's the overly friendly ones who are more likely to be up to no good. Sure, there are loads of men who are kind and helpful and it's hard to explain to a child the difference. But a molester is eager and approaches the child when the child isn't asking for any help. That's the difference. If your child asks someone a question like where are the towels or they can't reach the soap dispenser, then fine. But if a man starts talking to him first unsolicited, that's a red flag and he should know that. It's better to err on the side of caution and be wrong than to assume everyone is kind and well-intentioned and be wrong. A decent man will understand why there is caution.
It's hard. I have a son and a daughter, both grown now but it was difficult navigating these waters. I did NOT want them living in terror and afraid of everyone, but I also didn't want them to be naïve and therefore more vulnerable. The children who have no idea at all about potential dangers and much more vulnerable than children who have some understanding of these kinds of things. It's sad to take away a bit of innocence, but it's protecting them without being a overprotective helicopter parent, dragging them into the women's room when they're too old for it. That can cause other psychological trauma, so by avoiding one kind you may be inadvertently inflicting a different kind of trauma. You're giving him the knowledge and the tools he needs to navigate the world as safely as possible because you aren't always going to be there.