To demand Lord be removed from the panel because they’re non-binary is a bit like demanding the removal of anyone who is gay/lesbian from a tribunal panel
Saying you are non-binary is not the same as being gay. Even so, that is nothing to do with why Lord should recuse himself or be removed.
He should not be involved because he's a politician who has repeatedly and publicly campaigned against women's rights to single sex spaces and removed them during political office.
The protected characteristic in this case is religious belief. To clarify your analogy- it would be like Anne Widecome being a magistrate advising on a case where the plaintiff was allegedly sacked for being gay.