I think questions really need to be asked as to why his behaviour was remotely tolerated - that audio segment that was leaked was utterly vile. It wasn't even 'just' a case of him commenting "She's a bit of alright!" when an attractive person came on the stage; he sounded obsessively creepy in trying to analyse her potential 'use' to him with his mind.
I fail to believe that was a one-off, either. Yes, it's TV, with all the lights and glamour; but it's also their workplace, with a colleague who was behaving grossly inappropriately. I can understand that the other judges may have felt too awkward and even scared to speak out; but Simon would have had that power, if he'd been able to look beyond protecting DW and his own income.