No way would I want him going into a public men's loss on his own such as in a car park, shopping centre, etc at that age. Maybe OK if it's in a quieter/private place like a small cafe, museum, children's attraction, etc and I could see who was going in and out, or if it was one of those "single" toilets with a locking door in the smallest of places, but not "communal" loos open to the public.
Even OH, a big burly bloke, hates going into public mens' toilets because of the "strange" people sometimes in there and the general mess. He's sometimes gone in and come straight out again when he's seen the state of it or the state of people in there!
If there's no one waiting and no queues, etc., I would take our DS into the disabled/changing toilet at that age, and just make sure we were quick, just in case a disabled/mother&child came along needing it. When we both needed the toilet once, he went first, and then we came out to let someone else in, before we went back in for me to use it.
He started using men's loos himself when he started secondary, but even then, if OH was with us, they'd go in together, and we'd still avoid most of the public ones in open spaces like car parks and shopping centres!