He isn't really patrician-looking enough. He seems more like a very posh cuddly bear than a lord. When Downton first came about and I saw Hugh Bonneville was to be involved, I thought he would be playing a butler, like Carson.
I think that Ralph Fiennes or similar would have been better and I think he is the only bit of weak casting in the show. He just doesn't convince me. 