The problem with AGP isn't so much that it's a fetish, it's what it signifies. A fetish for fur, for instance, isn't harmful (unless it's real fur.)
Would I have a problem with a bloke practising his cross dressing fetish in the privacy of his own home? Mostly yes. If it was very unusual and confined to a small section, maybe not so much. But I don't think it is. I believe it's quite prevalent.
It's a product of harmful sex role stereotyping, as someone upthread said.
It's using the objectification of women as an arousal tool. If women weren't seen as merely sex objects, the sexual component to AGP wouldn't exist. It would just be the personal preference for the colour, cut, style, comfort of female clothing.
And AGP specifically is a nasty one. The fantasy usually involves the subjugation and humiliation of the person themself. They see women as victims and get off on assuming that role.
'Forced feminisation' and 'sissy porn' play directly into the idea that women are there to be used.
The jarring dichotomy of a man wanting to be humiliated and victimised, and then becoming aggressive and misogynistic if denied is both weird and scary to behold.