Isn't it true, though, that you could defend most any tax on the rich with this logic?
Whether or not you can defend it is less relevant than whether you can collect it. As people have mentioned the super rich tend to be good at avoiding tax and if push comes to shove if you tax an activity or possession too far people do something else or own something else and then there is no tax.
Equally, we live in a democracy, not 18th century france and well off people not only vote but tend to have influence.
However, the fundamental point is that you can only tax people with money to spare. You can call a single mum on benefits a scrounger as much as you like, but she still isn't going to have spare cash to fund the defence budget, libraries, mending of potholes etc.
If you are looking for people with spare money, dead people seems a fairly good starting point.