I suspect there are al sorts of scenarios that have taken place over time - but as a male I haven't seen any issues with accepting men of 'all persuasions' into male spaces - plenty of male sporting teams regularly dress up in dresses, men wear kilts in Scotland, if a bloke wants to wear a bra into a male changing room then that is up to him - he will probably have the proverbial taken, just as much as if he rocked up wearing cartoon underwear but there wouldn't be an issue with his being there...
I am sure there have been issues with trans-women in male spaces - just as historically there have been issues for some who are gay, but I don't think that reflects the whole of male society which in general isn't all that bothered (much like most women are not bothered by having trans-men in their spaces - all are women).
I am sure that the vocal level of concern in this area is partially driven by the need for it to be seen as an issue to have the excuse for trans-women to be in women's spaces (the nasty men were mean to us...) but I suspect that like much of the rest of that specific agenda there is probably a large amount of fabrication surrounding occasional truths.