The hospital ward thing is interesting. On single sex wards you get staff of both sexes working, who will see you in vulnerable, naked, emabarrassing bodies states.
Most patients are ok with both male and female hcps because their priority at that time I should to get the professional care they need.
I think the difference with sharing a ward with a person not of your gender is that they are not required to be professional, and they don't spend their working life looking at bodies, plus they are there with you 24 hours, often for days or weeks at a time. So staff is different to fellow patients. If you want to complain about medical staff you can do that and often the first response is that the member of staff is kept away from you.
Similar for prison staffing I expect, though I have experienced that myself.
To those saying they would be perfectly happy if trans men used women's spaces, I'm not sure that bears scrutiny. Trans men often pass more readily than trans women. Women are highly alert to men in women's spaces. Trans men are treated with hostility in women's spaces from fairly early in transition, and women don't hold back if they read someone as male in the ladies loos.
I don't think I would be ok with having a trans man in the next bed, or in a changing room. Which seems weird, but is true. Knowing that someone is biologically female doesn't seem to stop the part of your brain that reads gender signals from interpreting that person as male.