It's patronising. I wouldn't have liked it - there are connotations here which are not recognised when people dismiss it as "just humour" - the old boys club, shaking hands on the golf course, boys network that goes on at high levels of politics means this has more significance than your average conversation.
And that is what I'm sure a lot of people saying "I don't have a problem with it" are missing. They're thinking of themselves in that situation where a male colleague would say it to them; or some bloke in a shop; or a neighbour; or their husbands.
High level politics is a much more complex environment than the above situations, and our reflection on his comments shouldn't be taken lightly.