I don't think there's anything wrong with a 3 year old toddling around a cul de sac with 4 older girls.
I don't think there's anything wrong with a three year old running around naked in her own yard.
But if I were a man and a bunch of little girls I didn't know came to my house, and one of them were bottomless, I would worry.
One of the neighbors could be a cats bum mouth busy-body who would spread it round that half-naked little girls in his yard. And as other people mentioned, since he doesn't know the OP, how did he know she was aware of her daughter's whereabouts? She could be a crazy paedo-sighter for all he knows, and could come running up shrieking at him at any moment.
There is a girl in our neighborhood who would often go to the park with DS, DH, and I. (She was about 9.) She would always try to hold DH's hand, rub his back, ask him for a piggy back ride, etc. I told DH that under no circumstances should he take her to the park without me, because sadly enough all it takes is a rumor.
OP, YANBU to let your daughters out to play, nor are YABU to let your daughter play naked on your own property.
YABU to say this man has a "filthy mind" and to let your daughter wander around to neighbors she doesn't know with no pants.
IMO. :)