I was telling DH this and he really thought my fear was ridiculous. Yes, on a logical level, I know it was unlikely that anything bad was going to happen but on an instinctive level I just wanted to get the hell out of there! I don't think many men really understand this.
They don't understand this because they've been socialised not to listen to women but to dismiss us when we voice our fears, especially about them. As a society we're so invested in pretending that male violence isn't systemic, just random lunatics each time with no connection to each other or our society, that men simply don't recognise that every time women meet new men, we have to assess our safety with them in a much more conscious way than men have to when they meet each other.
Your DH's first response to you telling him of your fear, wasn't to question why men make women fearful, but to dismiss your fear as ridiculous. Because if he was worried in this situation, then it probably would be ridiculous, a) because he's able to defend himself and has a fighting chance of doing so effectively against any attack and b) because he's not in the target group of regular victims of attack by predatory males.
So because he's the default human, he assumes that what would be an appropriate response for him, would also be the appropriate response for you.
Men have been socialised to understand that with other oppressed groups than women, there is good reason for those groups to be cautious; he'd be unlikely to dismiss the fears of any other group on the receiving end of targeted violence than women; but because he's blissfully unaware of how normal the background to our lives of male violence is (and he's not listening when we tell him), he's not able to make the same leap of empathy that he would if someone from another oppressed group was telling him of their fears about being in this vulnerable situation with a member of the oppressor group.