"obviously mad then!"
not at all - it's awkward because while women dress how they want "for themselves" and not necessarily to attract partners, they may sometimes be perceived as wanting to attract partners (and for all I know the desired partner might be another female!)
Yesterday I was walking along, minding my own business and three twenty-something young women overtook me. One was talking about her forthcoming marriage, all were quite jolly after a shopping trip (by the number of bags).
One had a short black skirt on, down to hardly an inch below her bottom, and had to constantly pull it down. It was difficult to ignore, so I crossed the road, and no way would I say she was 'asking for' anything (attention, or being assaulted), but sometimes without wishing to, clothing will get car drivers' attention, wolf-whistles from a building site, unwanted verbal comments, or physical attention, and no way would the perpetrator be free of guilt, or the woman be 'asking for it' but one cannot ignore the clothing being an integral part of the situation.
Don't think I expect women to need to wear drab clothes, or be unable to show their natural features, but perhaps sometimes a couple of inches more fabric would make the difference, rather than showing bare midriff or a lot of leg or having trousers seemingly falling down...
It can even be embarrassing as an onlooker when (as happend in recent weeks) a young woman with her daughter was waiting outside the supermarket. Child walking about chattering to her mum. Mum (waiting for her own Mum to come out of supermarket) was sat on low wall, leaning forward. Trousers low cut, cheeks of her bottom uncovered because she had a g-string on. Was just feet in front of me while I was waiting for a neighbour to finish shopping so I could get a lift home. I looked away, but could have been classed a 'dirty old man' if I hadn't done so. Not fun to be put in that position without any wish!