It also annoys me when I'm walking along with my dogs, on the lead, perfectly under control, and we pass a toddler or small child on the other side of the road/path whatever who start wailing that there's big dogs.
It doesn't annoy me. Admittedly I've only had it happen once (most kids want to fuss our dog, he's a fluffy spaniel type!) - I saw a little girl was scared and crying, and her mom was saying, "It's OK, he's on a lead, he won't come up to you". I moved him as far away as possible on the other side of the path, got him to sit, and had a finger in his collar so there was no chance he'd move out of position. The mother said thank you, and they passed by.
Generally our boy is kept on a lead, except for certain quiet fields popular with other dog owners, where he's allowed off lead and we play fetch. Wouldn't dream of letting him off near children's play areas, not because he'd savage the kids, but because I don't know how the children would react to him. Would they be over excited and scream at him, or scared? Go up to him unasked, pull on his tail, poke him in the eye, etc? It's not fair on him or the children.
I don't know what past experiences people have had with dogs (I had an ex who was scared of dogs, in his case it was because he was bitten by a sausage dog at the age of eight in France, and had to have rabies jabs!), and even my smallish, very friendly dog could appear intimidating to a small child, given their height. My husband doesn't like being near horses or cows, for example, because they're large animals.
Even if an adult's fear is completely irrational, I wouldn't dismiss it. I don't like daddy long legs - I'm terrified of them. My brother is terrified of spiders. Both completely irrational phobias, and sillier than someone's fear of being approached by an unrestrained dog.