There are concepts which are called phenotypes. A phenotypical woman will have primary sex characteristics of a womb, ovaries, vagina, clitoris and vulva and a hormone profile in which oestrogen and progesterone are dominant and testosterone is secondary. She is very likely to have secondary sex characteristics of developed breasts, a high voice, a narrow waist and wider pelvis and hips, and no beard, as well as a shorter average height (and others which would take too much time to list). This is again very likely to result from a sex chromosome pairing of XX.
A phenotypical man will have male primary sexual characteristics of testicles, a scrotum, a penis and a prostate gland, and a hormone profile in which testosterone is dominant and oestrogen and progesterone are secondary. He will very likely have secondary sex characteristics of undeveloped breasts, a beard, a low voice, a narrow pelvis and hips and a higher average height. (Again, the full list is longer). These are likely to result from XY sex chromosomes.
Whether someone is a man or a woman (or, if a child, a boy or a girl) depends on which phenotype that particular person better matches. In most cases, people are pretty close to one or other phenotype and generally casual observation will suffice. That is aided by the fact that most people choose to adopt modes of dress and behaviour that are particularly associated with one or other sex.
While there are a number of recognised medical conditions that make those who have them a less good match for one or other phenotype, the number of those people is proportionally small (though a world population of 7.8 billion means that there must actually be quite a lot of them - there are different views in medical-scientific community as to what the proportion is, but if as a PP said, it is 1 in 40,000, then if there are 68 million people in the UK there are 1,700 of such people), most of them will identify more strongly with one or other phenotype and behave accordingly.
Ultimately, though, because biology is messy, there is no exact definition.