What's you're talking about is an argument called "reducto-ad-absurdum". People are taking a statement you've said, and using the same sentence structure have replaced certain nouns to demonstrate that the argument is absurd.
The problem is it's not absurd, what they've done is a kind of 'logic' sleight of hand; they have by changing the nouns also changed the context and thus made it absurd. This is also often called a "straw man argument": not addressing the principle argument they make it weaker by saying it is the equivalent of saying x, y or z and that's absurd so your original argument is absurd.
Saying that women should have access to women only spaces is the conclusion of the argument. It is is tied to "suppressed premises" (premises that aren't overtly stated) which are:
P1/ it is good for people to not be put in danger/oppressed.
P2/ people who are in danger/oppressed should be given refuge from their danger/oppression in places that exclude their oppressor
P3/ women are oppressed/put in danger by men.
C1/ therefore women should have access to women only places.
But yes, people will try and argue with you because 'poor trans people feel like women'. Fah- feelings don't make for arguments or truth, but generally make for nonsense. By pretending the above argument is sexist or racist makes absolutely no sense now because if you replaced the words women with white or men or some such you would have absurd premises (such as 'P2/ Men are oppressed/put in danger by men').
Thus they actually defeat their own straw man argument by the same principle...interestingly one could make the argument that trans people should be given their own spaces by the same logic- to which I would actually agree. But it is fallacious and wrong to assert that women's spaces should be made available to men (by which I mean biological men and women, nothing to do with gender identity).
And that is all ithe argument about giving up women's spaces is: an assertion based on emotion. I am not bound to agree with an assertion someone else makes because they feel like it. That way madness lies.