This is where language is important.
"But I’m not policing rhe toilet: it is for women and trans women, I’m not worried men will entre (sic)"
Male people = those without a gender identity and those with a gender identity.
Female single sex spaces are not for 'women'. They are for female people. Not for male people. Using unclear language removes the facts that are pertinent to clear communication.
It is perfectly ok for you to use this language. What is not ok, is for any person to shame or demonise people who choose to use precise and accurate language. And that includes pronouns too.