If I heard a woman say 'don't touch me' to a man, then I would assume it was because he was a man and not on the basis of his colour (or he was known to her and they had some history, or he had copped a feel or something like that).
He made the assumption, rightly or wrongly, that she didn't want to have physical contact with a black person, and kicked off on that basis.
If a woman said it to another woman, then I may wonder why she didn't want to be touched be her in particular. I may think it could be race, religion, class... or even a bad case of sunburn/lumbago being pressed against.
It happens all the time on London transport - people barge into you, try to walk through you, almost sit on your lap, whack your bags, wallop you with their bags and rucksacks... people are cross and aggressive too, and if its happened to you for the umpteenth time that day, yes tempers are getting flared. If lucky, people just give each other a nod or smile, or 'sorry mate'. If unlucky, they will start muttering, cursing or scowling. And yes, some buggers do shoulder-barge you on purpose to get a rise.