There has been a fair bit of talk on this thread about how a man dressing as a woman would likely be the subject of ridicule. Or how a crossdressing man could be the victim of violence (harassed or even beaten up in the street).
I think the above is key to understanding why some of us on this thread find crossdressing offensive. When a man crossdresses he takes on clearly identified trappings of femininity and 'embraces' the performance of femininity; being 'girly' (rarely 'womanly'), giggling, hair flicking, tottering about in high heels, wearing restrictive clothes that are designed to display a very contrived definition of female sexuality and present the wearer as something to be looked at.
The trappings of femininity vary from culture to culture, but they are all there to perform the same function; to 'other' women. To clearly communicate that men are the default and women are 'other'. To show that women are the sex caste. To show that women are decorative objects. To show that women are below men on a hierarchy. To show that men are in charge and men make up the rules. To clearly show that society is made up of two distinct groups - one of which dominates the other.
Feminist analysis (and books like Beauty and Misogyny which I mentioned earlier) examines femininity within the context of society which is founded on a system of social organisation which firmly and universally places men above women. Gendered clothing and all other manner of beauty practices are part of that system. The trappings of femininity are the gender equivalent of the yellow star. They are there to tell us what our place is. They are there to differentiate the sex caste from the ruling class. And the fact that some women may like the trappings or enjoy wearing them, does not change this one little bit.
Therefore, it is offensive to many of us when a member of the default class takes on these trappings for fun and giggles. (Someone up thread called it gender tourism which is a really good analogy.)
So back to why a crossdressing men may be the subject of ridicule and even violence. It is down to pure misogyny probably with a good dose of homophobia thrown in. A man dressing as a woman is lowering himself. He is doing the worst thing a man can do - act like a woman. Because everyone knows that women are ridiculous and 'other' and objects of contempt and the sex caste. A man acting like a woman is the object of contempt by many other men because he is acting like a woman - like a bitch, like pussy, like a sex object, like a slag, like a wife. (And he may well be homosexual which is just as worthy of contempt because it is pretty much like being a woman).
It may be fun for a man to pretend to be a woman sometimes and fetishize the objects and beauty practices that are our yellow stars. But as another poster said upthread, some of us have to live here. This is our reality. This is our place. To take things to an extreme for the purpose of discussion - what would we think of a man who enjoyed binding his feet and showing off his tiny (deformed and extremely painful) feet to the foot bound girls and women of China? Who fetishized foot bounding and found it an expression of 'his feminine side'? Who found it pleasing and girly and part of what makes a female human being a woman?