who just quietly and harmlessly enjoy aspects of this in private
No. I think they are generally coercing their wives into participating in it and so it is not harmless.*
Why would you presume that? Is there any reliable data on the number of casual cross dressers who just do it in private, and what their partners think of it? Personally, I find it somewhat arousing myself, though that's partially because I'm quite attracted to a male androgynous look. I'll buy him items occasionally and occasionally request he wear them.
Obviously there are coerced wives, like with any other sexual niche/fetish, but I think a lot of people assume that there must be coercion because it doesn't appeal to them/they'd find it a turn off, rather than because of evidence.
Sexual fetishes don't belong out in public though - and cross dressing shouldn't be an exception to that. Those exposed to it should be freely consenting adults only, no exceptions.
Whilst I want to break down 'gender' as a concept, and people be able to wear anything they want, there's a world of difference with a guy wearing a skirt because he likes it, and just fancies wearing it, and one that is wearing it to involve bystanders in the fantasy of being a woman. The difference is usually material, tightness and several inches, but it's really obvious, even though on paper it doesn't seem to be 😂