I think it is because it is actually quite difficult to justify. Rather like it being legal to place male rapists in women's prisons. A man potentially winning any medal at a female event is an uncomfortable, stark reminder that women do lose out.
The thought process is 'of course TWAW so they must play women's sport, room for everyone, let's all be inclusive'
'But hang on a minute, what if that means two men in the ladies final at Wimbledon'
'That will never happen, such a small number, would never do it for advantage, blah blah'
'Look - there is a man in a female event at the Olympics. This just shows what could happen'
'Mmm - not a mainstream sport, though, doesn't matter, they didn't win'
'Yes but could you guarantee that a future Olympics won't have a line up of all men in the women's 100m final? How?'
'Ummm, errr, inclusion, science is still evolving, not a clash of rights, nothing to see here'