Well, I can't speak for everyone else, but for me, I am wary of (not hating, drama much?) Transwomen because they are often motivated to transition by AGP, and I don't feel comfortable around men who get turned on by playing out an over-sexualised, insulting idea of womanhood. Especially as many transwomen with AGP also find transgression of women's boundaries part of the turn on - see wanking on knickers in the m&s changing room etc.
Transmen usually don't engage in aggressive destruction of men's boundaries or single sex spaces. Nor are they usually motivated by sexual fetishism.