You're suggesting if someone is gay or trans they are above reproach, and somehow exempt from comment on their safeguarding practise?
Accusations of homophobia and transphobia are an insult. Gay and trans people are just as capable of acting appropriately and respecting boundaries as anyone else.
His sexuality is irrelevant, as far as I've seen. This is about his communication, especially to young and vulnerable people. His behaviour is encouraging vulnerable people to take risks, lower boundaries, reject their family.
It mirrors or echoes dangerous behaviour that a lot of women recognise as a manipulative, coercive dynamic.
Jeffrey may just be very bad at communications. And not have a good awareness of boundaries.
In the end, I don't know how much his intent matters - it's the effect that it has. And women are going to keep on pointing out red flags when they see them.