I don't see a problem with members of a religion holding a conference in lines with the teaching of their religion.
I see a massive problem if any criticism of their religion, because it it is misogynist, homophobic, and encourages violence, is not permitted because it is inciting 'hatred'.
I also see a massive problem if an organisation which promotes fundamentalist interpretations of this religion, and supports terrorism against the West, and which is banned in many countries, is allowed to operate in the UK.
I see a massive problem if members of this community are allowed to repeatedly marry their cousins, which increases the risk of severe congenital abnormalities in their children, but fight any suggestion of banning it as an attack on their 'culture'.
I see a massive problem with members of this religion threatening people who 'disrespect' their holy book, including making death threats against a child, without any sanction.
I see a massive problem with members of this religion threatening to riot because they don't like a football team, and rather than ensuring sufficient force is available to enforce the law, the football match is cancelled.
I see a massive problem with women from the community being allowed to take part in any kind of public life while covered head to toe in a shroud.
I see a massive problem with MPs manufacturing a non-existent terrorist plot against this community, while failing to address the huge and real ongoing terrorism committed by this community.
I see loads and loads of problems with this religious community.
But a men only religious conference, meh.