Sort of! Without knowing what this person threatened to sue you for it's tough to say, but lots of people 'threaten' to sue with no real intention of doing so and it can be scary (especially if they do it via a legal firm), irrespective of it being an empty threat. So you were right to take advice, and the advice you got may have been good...or it may not.
Liable is an especially tricky one...you certainly don't need to be 'Elton John rich' (which is a term I'm definitely stealing!) to see it through. We've all seen the big celebrity cases that make the news, but I've known tiny businesses successfully sue for libel over lost custom/reputational damage.
The UK legal system is incredibly favourable to those bringing cases of libel/slander. One of the most favorable in the world. For example, if you're in court accused of stealing my car, you don't have to prove that you didn't steal it...the prosecution must prove that you did. Now with libel, if I take you to court accusing you of making a defamatory tweet about me, it's up to you to prove that your tweet wasn't defamatory, with a limited number of defences available.
When you factor in the costs involved in defending a case that goes all the way, you can see why so many people are advised that settling is often the better option (especially if you're bang to rights like OP seems to be).
As for the "they wouldn't get anything if you aren't rich" argument - big disagree here. If that was the case, poor people would have carte blanche to libel others. If someone has the resources to take you to court for libel, assume they have the resources to pursue you for the money if they win.
Attachment to earnings, funds from the sale of property, bailiffs...all these options are open. Being forced into bankruptcy is no fun!
More to the point, the stress of any/all of the above for most people can be life changing. I wouldn't suggest to anyone that they could get away with libel because it's expensive to bring or that they aren't Elton John rich!