Some of them may well have disabilities, some caused by surgery, but yes, it's a potential sticky wicket.
Particularly since we're not supposed to say that there are mental, hence invisible, disabilities involved, which might well entitle trans people to use those facilties, and they wouldn't want to admit that either.