It's not meaningful to ask whether a single parameter (like sex) 'is' binary, bimodal, or monomodal: it's a question that can only be answered by measuring another parameter.
With respect to sex (as registered at birth), the following have binary distributions: karyotype; possession of testes or ovaries; testosterone levels (also Q angle, neck circumference and grip strength have negligible overlap).
Bimodal: height; strength; speed; criminality.
Monomodal: height (in certain populations, with a flattened curve); IQ (with differing amplitude at the median point).
And, before anyone starts, XY women with testes, or with no gonads, are very rare, and abnormal (in the statistical sense: it's not a pejorative).
An alien would have no trouble discerning sex in humans.
Once you know who the men are (XY: normal development), then the fact that they are likely to be stronger, and more violent, than any woman that they meet (and potentially able to impregnate her) shapes the whole of society, and the rules that it adopts, whether to safeguard paternity, or the women themselves, and irrespective of the existence of some weak men, and violent women.