I don't think all trans women are perverts but an awful lot of them are. I don't think any of them should be using womens spaces. The problem really took off when Stonewall shoved all categories under the 'trans' umbrella. We all knew a transsexual was different to a transvestite or a drag queen.
Duncan Bannatyne is one of the few who has, once he became aware of this, ensured that the changing rooms in his facilities were based on sex not gender, so I do feel it's a bit unfair to pick on him. I think he's accurate in what he says and fair in what he does.