Ah yes, I'm glad you raised that. It's worth exploring.
It is factually correct that gay men present the same risk towards vulnerable groups as heterosexual men do - neither more nor less.
It is also factually correct that lesbian women present the same risk towards vulnerable groups as heterosexual women do - neither more nor less.
So you are quite right that if you are worried about protecting vulnerable groups from sexual assault then there is no need to discriminate on grounds of sexual orientation - gay or straight makes no difference. Only prejudice would insist otherwise.
Further: It is also factually correct the risk towards vulnerable groups presented by men is higher than the risk presented by women.
Now - can you see the difference - and the problem - yet? Transwomen are men who want to be treated as if they are women. Some of them believe they "really" are women. They want to be risk assessed as if they are women, they want to be treated as if other people are just as safe around a transwoman as they are around a woman, they want other people to believe that people are just as safe around a transwomen as they are around a woman. But the facts say different. It is factually correct that transwoman present the same risk to other people as men, not as women. So it is not prejudice but wisdom to treat transwoman as more risky than women. It probably wouldn't be fair to treat transwomen as being more risky to vulnerable groups than men, but honesty and evidence means we have to accept that transwomen present the same risk to other people as other men do.
This is very different from gay men and homophobia. Gay men don't want to be perceived as lower risk than other men, they don't want anyone to pretend they are different from heterosexual men in this regard. Gay rights activists just accept that gay men should be compared to other men. Transactivists aren't willing to accept the same comparison.
Caring people often worry more about the reverse problem - about how vulnerable transwomen are to being assaulted by men. Transactivists use this (with whatever degree of cynicism or genuine belief) to justify transwomen using women's spaces. But transwomen are not at greater risk of sexual assault than women are, though they might be more at risk then other men. And whether they are at more risk or not, it is still quite possible to be both extra vulnerable and extra dangerous to others (think of Lennie in "Of Mice and Men"). Transwomen remain a risk to women, whether or not they are also at risk from men. It is not prejudiced or transphobic to recognise that.
Transmen are a slightly different issue. Huge doses of testosterone can make women more sexually active and also more aggressive which could make them more of a threat to others; but I don't know how the numbers stack up in reality.
(I like the idea of a "card carrying bisexual" btw. How did you earn your card?)