The reason they think we are transphobic is they actually believe that being trans makes you the opposite sex. They have fallen so far into it that it doesn't matter that the transwomen has male anatomy, when they called themselves "she" they actually believe they turned into a woman and therefore are no greater risk to women in our spaces than other women.
It isn't transphobic to say that trans women arent women. Biologically its a fact that can never be changed. It is no less accurate than me telling you Im actually a hedgehog :-).
A transsexual person can be gender female but biologically will always remain male. Surgery is an adaptation to the male body to allow it to socially resemble that of a woman but it doesn't make that person biologically a woman.
In terms of risk though I do think it is important to differentiate between transsexual and others who identify as simply "transgender".
A transsexual person is medically assessed and diagnosed with gender dysphoria and transitioning through hormones and likely surgery. Such a person IMO does not represent a significant risk and transsexual people want to get on with their life quietly and peacefully. A transsexual person will be the first to tell you the TW are not women!
A transgender person (ie under that umbrella but not transsexual) I accept does pose a potential risk. Typically they are not medically assessed, not medicated through hormones and in some cases sexually motivated to "transition".