Based on my life, I can say more times than not 'sex affecting how drugs work in the body' refers to the primacy of sex hormone affecting tissues in the body and their processes rather than the state at conception or birth or however people determine sex.
Medical personnel know that the younger a person begins HRT, the more likely their cells and systems will conform to normal ranges of the adopted sex. Blood tests are a simple example, body composition ratios are another.
Some trans women end up learning they have acquired new autoimmune disorders thanks to how cells and systems in their body react to being estrogen dominant. Some trans men experience a lessening of their EDS, RA or Lupus symptoms as soon as they begin testosterone.
I informed the surgeon and thusly the medical team of my birth sex when I had recent abdominal surgery. He said it wouldn't change anything as all the applicable tests performed were all in female range. I'm sure there were some practicalities of the actual surgery itself provided my organ distribution and maybe placement. But medicine and chemical reaction wise, all went as expected. So in this situation, 'sex' does not appear to be dependent upon gametes.