Many of us have already agreed that we don't see all trans people as fetishists, but that some people claiming the label trans are.
At the moment we have laws (the Equalities Act, the Gender Reassignment Act) under which the protected characteristic in law is gender reassignment, not gender identity. In theory (if not in practice) gender reassignment is subject to gate keeping - among other things, you have to convince a psychologist that your motives are genuine and (this is based on what transwomen themselves have told me) specifically the process is designed to weed out men pretending to transition because of sexual fetishes.
So we have a legal framework on the books right now which attempts to balance on the one hand the rights of trans people to live fulfilled lives as they see fit without interference with others and on the other hand the rights of women to have single sex spaces and provision where that is necessary and proportionate (the Equalities Act is absolutely crystal clear about this). Like all legal frameworks, it doesn't work perfectly, but it is at least taking into account the possibility that the needs and rights of women and trans people are different and may on occasion come into direct conflict, conflict which needs to be resolved in a reasonable way, not by screaming "transphobia" and "no debate".
There is a push to replace this with self-ID, which would provide no such mechanism to keep male-bodied fetishists out of women's spaces. If you are in favour of self-ID you owe the rest of us some account of how you're going to deal with the fetishists. You owe us an account of how you're going to handle the Karen Whites, the Lauren Jeskas, the Lisa Hauxwells of this world. You owe the woman who turned up for her smear test to be confronted by an obviously male HCP saying "but I am a woman." You owe the women who love playing rugby an account of why they should put up with a massively enhanced risk of spinal injury when the opposition decides to play a 6'4" male bodied individual in the front row of the scrum.
You cannot just change the law on the basis of less than 1% of the population while materially affecting the lives of 51% of the population.
This is not like gay marriage where what a couple chooses to do or not do in their own life has no impact whatsoever on anyone else.
This affects whether I as a woman have the right to ask for a female HCP, undergo rape counselling in an all-female support group, flee from DV into a shelter I know will be free from male-bodied individuals, be imprisoned only with female bodied people, not have to get naked in front of a stranger with a penis when I want to go swimming (I can think of three pools I use regularly which only have open-plan single sex changing rooms), not play contact sports against a male-bodied individual who may cause me lasting spinal injury. And it affects every other woman in this country in the same way.
You don't get to silence debate on this by shrieking "transphobia" at every opportunity. Nor do you get to tone-police women who are justifiably angry at the way a well-organised lobby group has been attempting to sneak these changes in under the radar without anyone noticing.