There is indeed nothing wrong with being trans. Identify how you please, dress and refer to yourself how you please. That in itself is no threat to anyone (other than maybe to themselves if they find themselves in a situation with people who will react poorly to them), however, trans identifying men who insist on using female only spaces (toilets, changing facilities, domestic violence refuges, breastfeeding groups, hospital wards, prison wings), absolutely are a threat to women going about their every day lives.
Even if they themselves present no threat to the women in question, their mere presence gives the impression that there is (or should be) access for other males who may be harmful to said women, who are not genuine or are only there for nefarious reasons, therefore essentially a threat to women.
Thank goodness for the recent clarification in law, that sex is binary, men are men and women are women, the law supports this, and single sex spaces are exactly that!