Our brains are plastic (in the sense of changeable). So it makes sense that how you think, which is influenced by socialisation, would impact your brain. It makes sense that the experience of being in a male, or female body, would influence your brain.
But from an evolutionary perspective, it would seem wasteful for a species to develop two different kinds of brains. So brain differences are surely post-fact, not intrinsic?