I haven't looked into intersex super extensively to preface this, but from what I understand it's generally seen that in the case of intersex individuals, as they grow up and go through puberty their body naturally veers toward showing more characteristics of one sex over the other.
I believe that's where the term "sex assigned at birth" originally came from before it got warped into applying to everybody.
Originally doctors would make a decision based on the intersex baby's genitalia (which part seemed more dominant to them) what that babies sex is, and typically surgeries would be done when to align with the doctor's decision in the name of giving them an easier life.
In that circumstance specifically I could understand that perhaps literally having a doctor assigning a sex to someone who is clearly in between and taking surgical measures to "correct" this, probably shouldn't ever be done. Especially to a literal baby that obviously cannot consent.
Mistakes can happen, and I feel like even a doctor can't realistically know one way or the other in that ambiguous context until the person has gone through puberty.
At which point I feel it would be fairly obvious that the individual was probably genuinely "mis-gendered" at birth for want of a better word, because their body would show that from how their puberty manifested.
Does that make sense?
In the opposing case where lets say their puberty veered more towards male than female and they still decided to change gender, I'd be probably a bit confused tbh?
And a bit concerned about size difference/strength difference/privacy etc still, but for the most part I just think everyone should live their life how they want as long as it doesn't harm anyone else.
So I guess it kind of depends on how puberty manifested itself for them honestly, I think that's the only somewhat fair way to approach it because it's puberty that creates or exacerbates the differences between males and females.