Even if you think are raising your children gender neutrally, the influences are there in the outside world.
Strangers on the street used to stop me and want to discuss how I was "in trouble" with my boy, but I'd be relieved when they were a teenager, boys need so much exercise, they remembered how their boys were exactly like mine, their girls were so much easier.
Problem is, DD is a girl. She got constantly told to her face, that no, she wasn't, she was a boy. She used to get stopped on the street and asked if she's been at her sister's clothes, and not in a nice way. She even got harassed (as a 2 year old!) in a swimming pool once because she was wearing a one-piece costume. They pick this stuff up and next time, they want to wear something different because they remember the mean comments.
Strangers will praise your boy for "boy" behaviour, and react negatively if they display girl behaviour. It's pervasive and endemic. Dress your boy in a pink t-shirt for the day, and you'd be surprised how peoples expectations change, as pp said.