For me, it's because their role is generally to determine medication and oversee the management of the patient, rather than to offer much in the way of talking support. Also because they are people who chose the path of becoming a doctor, which s typically about the accumulation of theoretical knowledge rather than developing the capacity for therapeutic relationships, or for offering compassionate day-to-day care.
Nothing wrong with any of that. It's just horses for courses