It’s an interesting question and it has an interesting answer!
The reason is because sex is not strictly binary when it comes to the human body. All human embryos are not formed to one sex, and actually an embryo starts to develop with everything required to be female. It’s not until much later in the pregnancy when hormones are formed that then start to adjust features. This is why men are born with nipples, because the body was preparing for breasts, but without oestrogen they won’t form and remain dormant.
A body can display different sexes. For example you might be male but have XX chromosomes. You might be female but have XY. With intersex people, you could be born with a womb but also a penis. Most of the time it’s pretty clear and you can just observe it as you say, but sometimes it’s not possible, sometimes a doctor has to decide the sex of the baby for the baby, taking into consideration all the evidence they have at the time. If a baby is born intersex, the doctors will make a call on which way to assign the sex, and perform surgery that they think will make it easier for the child as they grow up, by removing body parts that don't line up with the sex they chose.
Recently a lot of intersex people have said that they wish they had been able to make this choice for themselves, because they obviously didn't consent to it as they were only a few days old. Malta is the first country in the world to respect this law, so now intersex people can grow up with all their body parts intact, and then when they are are consenting adult, they can choose to have surgery.
So because although most of the time it’s pretty easy to tell, when you have lots of different evidence that conflict with each other, a person has to make the decision: hence it is assigned and not observed.
Hope that makes sense!