" think men invade each others personal space less."
I don't think that's necessarily true. Not in my experience anyway. They might not remove a stray hair but ruffling hair, clapping each other on the back, patting someone's stomach if they've put weight on.
I haven't got any axe to grind because I wouldn't even notice a stray hair and, if I did, I wouldn't give a shit that you had one on you. But I do tend to think that those people are trying to be nice. Not judgy, not mean-spirited, not trying to highlight your failings or criticise your appearance or invade your personal space. They just notice, and think you'd like it removed, because they would if it was them maybe. I don't understand, at all, why people are so willing to assume the worst possible motives in other people.
If anyone sees a stray hair on me, or a label sticking out, or a drip of snot on my nose, or my skirt tucked into my tights, or an item of clothing that's inside out, then please just sort it out or tell me, I'll thank you for it.