Most males aren't colour-blind but almost everyone who is colour-blind is male. So opticians look out for it more in males than in females.
Sure, but colourblindness affects about <googles> one man in 12, and one woman in 200. In other words, it is not a rare condition. When the base rate is high, it does make sense to use shortcuts like this.
To give another example, women know that a substantial number of men commit (sexual or other) assault, whereas very few women do. It is therefore not "misandrist" for women to be cautious around men, and most men understand this.
My point is that this logic breaks down when the event is very rare - much rarer than colourblindness. Even if 60% (let's say - I really don't know) of terrorists right now are brown-skinned people[*], terrorist attacks are sufficiently rare that it makes no sense to pull every brown-skinned person in for questioning after a bomb goes off.
[*] Back in the 1970s, terrorists were Irish, but very few Irish people were terrorists. However, the public and indeed the police did not really understand this, and Irish people were subjected to all kinds of harassment, up to and including being fitted up and sent to prison, because of a feeling that "Well, maybe this Irish person isn't actually a terrorist, but I bet he's friends with a few".