I think he looked so ill because of the stress and lack of sleep.
And yes, his reputation was ruined because he refused to behave in the way society expected him to, by divorcing his wife and announcing to the world that she was a cheat. Her beauty and ability to manipulate men in particular meant that no one could believe she would behave in such a way - and that of she hadn't, then he must be (although interestingly the women seemed more clued up on the real story - but of course no one asked them!)
His helping Miss Wannop and her friend at the beginning, and taking Edith back to London to help his friend, whilst motivated by gallantry alone, we're easily interpreted as a sign that he was being unfaithful. And then his brother and father relied on rumours and vindictive gossip rather than asking him straight out :(.
He's basically the last good man, an anachronism a bloody sexy one.