In the event that this goes any further you might want to start preparing your own complaint that your colleague is not respecting your protected gender critical belief that sex is real, is important and is immutable.
Unless you have expressed your views in a way that a reasonable person could find objectionable, I believe you are entitled to at least question whether somebody who is male should sit on an all female panel, or that at least people ought to be aware that the person in question is not a woman by birth. (I'm not sure if you are saying this is meant to be an all female panel or not, but I think people ought to be at least aware whether IWD panel members are men or women).
All this was decided in the Maya Forstater case which you can find a summary of in Wikipedia (although nothing on Wikipeadia is 100% reliable) Forstater v Centre for Global Development Europe - Wikipedia
To just quote quickly from Wiki, Forstater won her case "... after the appeal tribunal concluded the belief that "biological sex is real, important and immutable" met the legal test of a "genuine and important philosophical position", and "could not be shown to be a direct attempt to harm others." As such these beliefs were afforded protection under the Equality Act
I would also echo previous advice to contact Sex Matters and you could alos try the Free Speech Union. (Note they are not a workers' union, they are a group that supports the principle of free speech)
NB - I'm not a lawyer and I'm not qualified to give you legal advice. I'm simply stating my opinion based on a layman's understanding of what i think is the law. Hopefully this complaint will go no further and you won't need to do anything. However, if it does become a problem definitely contact Sex Matters and the FSU, and you might need paid for professinal legal advice.
But if your employer has any sense they won't let this progress