The mere fact that they are off label doesn't mean much. People take off label meds all the time.
The issue is more that you're trying to do the exact opposite of its licensed purpose, and stop a normal puberty.
It's a situation where you know the original safety assessment is pretty much meaningless; that it would need a new assessment with an assumption that it could very likely not be a good idea to stop someone's normal development.
And of course also, as mentioned, the lack of evidence that the treatments are really in a patient's best interest.
We don't have controlled randomised trials. We may be preventing some children from naturally recovering from gender dysphoria. There is also the issue of outcomes of transition vs trying to live with gender dysphoria in a different way.
For adults, there may be various situations where imo just let them sign a disclaimer for off label drugs and take the responsibility away from the doctor.
A somewhat common situation in the UK would be people wanting the antidepressant Bupropion which is off label in this country, and so GPs really wouldn't like giving it to patients, despite the fact that it does have advantages including the side effect profile.