It's his money, not the government's. He can decide what to do with it.
What difference does it make how rich he is?
If Labour were so worried about tax avoidance then they could have closed the loopholes and simplified the tax system.
They chose not to - indeed, they doubled the length of the tax code, making it vastly more complicated - and so people, perfectly understandably, and perfectly legally, take advantage.
My preference for a tax system is low taxes, few taxes, and very few exceptions - making avoidance hard. Instead Labour went for high taxes, many taxes, and gazillions of exemptions and special cases - making avoidance easier.