If the sign on the door is the only thing that makes the space single sex, we know that space is unlikely to actually be single sex.
If no one can be challenged, any man knows he can enter.
Its impractical to say that only members of staff can challenge and only if they are 100% sure they are talking to a man. There are so many loopholes in those conditions. Staff arent always around. A man could expose himself or attack women and girls in the time it takes to find one. 100% sure means that lots of men will fall through the net.
Once its known that the only safeguarding is the sign on the door, it will be a magnet for men.
Feeling sorry for the women challenged and understanding how insulted they feel is normal, but removing a layer of safeguarding as a result is not proportional.