"Women have suffered sexual microaggressions (and macro ones) at the hands of men for thousands of years. All of them, from what I can make out, by all of us. And I don’t want to run the slightest risk of being seen to have any part in any of it any more."
This is a telling paragraph, isn't it?
Women have been the subject of sexual aggression. And what does Giles think about that? He is worried how he looks to other people. He doesn't want to run the risk of being sexually aggressive towards women, not because of how the women might feel, but because it might be seen badly and it may reflect negatively on his career, the poor lamb.