mocked the idea that any man was actually taking advantage of access to female spaces
I've been privileged to do a decent amount of housing support work in women's DV shelters in more than one country. Only allowed because I'm a woman. And I get to have the hidden address, sworn to secrecy, not allowed to submit expenses that show location, given one phone number to contact, not allowed to share location with anyone, seen the walls, security measures, the care that's taken. I've also heard the stories about what can happen when the address is leaked.
Your ex doesn't understand that predatory, violent men will do anything to get their hands on vulnerable women. If they will work at it, train to get jobs that allow it, follow people for years, stalk and lie and chase and do all that, why on earth wouldn't they just say, "I'm a woman"? It's completely irrational to suggest otherwise. In a small town with one shelter, it would be as easy as anything to gain access to an ex.
Your ex is a bit thick, right? It can't be the only sign of it.