I believe it is down to our need to assess the potential risk in situations in order to decide which response is needed.
So when an individual is walking towards you down the street or whichever situation you are in you make a snap decision regarding how you believe they may act, for example: a tall muscular and aggressive looking male might be more intimidating than a frail elderly lady and thus prompt a different reaction from you.
Appearance is all we have to go on initially but when we develop more knowledge about that individual our opinion often changes. It's annoying if you feel you are being unfairly judged but unfortunately it's something that everyone does, whether consciously or not. It's a useful tool to keep us safe and allow us to act appropriately in certain situations but it's not always correct.
I found this to be the case when starting a new project, I was immediately drawn to a lady of similar style as I felt we would have more in common. We don't and actually we don't get on especially well at all.
It is the same with the bias we form from personal experiences and social influences growing up but I do think it's also important not to let it drive you to treat others unfairly and to always challenge your views.