Sexual assault isn't usually about being 'turned on' by the prospective victim. It's not like rapists etc choose their victims based on whether they 'fancy' them or not. When a guy exposes himself, for eg, he'll do it to any woman, regardless of what she looks like. He, and rapists, and assaulters, are far more likely to choose victims based on whether they look easier to overpower (which is why you get so many children being attacked). So that you and your friends aren't turned on by the idea of women in changing rooms is irrelevant.
Your example of being attacked by someone of another ethnicity is also irrelevant. Women who are attacked, assaulted, beaten etc are attacked, assaulted, beaten by men. (Yes, there may be one or two examples of where a woman sexually attacks another woman, but so rare.) Lots of men would be attacked by other men, of all kinds of ethnicities. So the fact you don't have a fear of being attacked by another person of the same ethnicity, is not the same as a woman's fear (or lack of) of being attacked by another man. Because statistically, she IS almost 100% likely to be attacked by a man... if she is going to be attacked or assaulted.
I appreciate a man speaking up about this, though. I've been trying to think of a way to equate the experience of the threat women feel from men (when out jogging in a park, walking in the dark etc) to a man's experience, but I just can't find a way to help men understand... being aware that if you are going to be attacked it is almost certainly going to be a man is just part and parcel of being a woman, in today's world. Because it's just a fact. So our concern is based in reality.