The desire for FAAB only conferences/meetings came after some transactivists hijacked a women only session at FEM 11. That had been a men and women all welcome 1 day conference in London. Some feminists wanted a women only space and so one women only session was run.
There was/is a divide between feminists, such as those who run UK feminista, who want the feminist movement to involve men and women, and generally more radical feminists who believe that the gains of the women's movement (eg WA & RC) were through the efforts of women only organising.
Back to FEM 11...this one session, an attempt to discuss the value of women only organising, which had proven to be successful in the past, was invaded by some loud transactivists who spent the session shouting out, harassing women, not letting others speak, behaving in an abusive and threatening way. Many women were left feeling very uncomfortable and no one as able to discuss the issues at hand.
Off the back of that and through the desire of many feminists to meet in a safe FAAB only space the radical feminist conferences have emerged. Femifest is the third of these in London, each of which has drawn controversy.
In a free society, which we are supposed to be, I don't see any problem with a self defining group meeting amongst themselves. I don't think that's a trans phobic thing to say. I imagine if a transman tried to get into one of the men only golf courses he would be promptly removed but no one takes issue with them.