Well yes, it is about excluding men from women's single sex services and spaces, because many women need those to be single sex for them to be accessible and to meet their needs.
You cannot meet women's needs and let men go wherever they want. The two things cannot exist in the same space.
There can be additional spaces and facilities where possible, but men (and activists) have to accept: women have rights too and men can't have everything at the expense of women. They cannot take and enter everything. There has to be equality of provision and consideration, equality of right to accessible and comfortable spaces, and this means that the women who need single sex spaces have access to them. And that men cannot go in those spaces.
The sticking point really, in all this, is still saying 'no, you have to accept that they get things too and you can't have it all' to men. Which shows really exactly why women need those sex based rights.