I don't care whether it's a "job" or not. What I care about is whether he's paying his "fair share" of tax. It's employed workers who are currently shouldering more than their fair share of the tax burden with student loan repayments, frozen tax thresholds, increases in NIC, workplace pensions, tapering away of child benefit, etc etc.
Those living on "unearned" income as defined by HMRC, such as rental income, dividends, interest, trust funds, etc pay less tax overall, because those "unearned" incomes aren't liable for NIC and those "unearned" incomes are more flexible and can be shared amongst family/spouses etc to save tax by using otherwise unused personal allowances, basic rate bands, etc., which an employed worker simply can't do.
So, yes, I would judge someone who's basically living on "investment" income, either from their own efforts or inherited or trust fund, because they're not paying as much tax as an employed worker. If the guy in question has an income of, say, £75k, his contribution to the country is a lot less than someone earning the same £75k as an employee. And that is completely wrong. It's almost a certainty that the guy in question (or whoever set up the trust fund for him) have taken steps to minimise the taxes payable - by lots of different ways that aren't open to an employed worker.