I don't believe in only considering the legality of the situation: ethics, rights and morality form my overriding considerations usually.
It's just in this specific situation the conflict of interest is:
perceived (but not actual) entitlement of the homeowner to reserve and keep empty the parking space outside their house for themselves
vs
actual entitlement of the public to park legally and considerately in a public spot
I feel in this situation it is morally wrong to expect to impact a driver significantly financially to preserve a false sense of entitlement to keep a public parking space unused.
Further, I think its a good thing for a driver to keep traffic congestion out of a city centre and walk some distance.
If I arrived at a parking spot at the same time as an owner I would certainly cede if requested. But I don't think every residential street in the country should be kept empty of parked cars to keep spaces reserved for the residents directly outside their homes, no.
I understand the inconvenience of the resident, having experienced it myself, but I have never blamed the owner of the parked car for exercising their rights, rather the decision makers who determined that my house didn't deserve a restricted parking spot.
There is a first come first served situation in much of life, and if the person before me in the queue gets the last item in the shop, I don't resent them for not putting my perceived greater need before their own, but I might resent the shop for not planning to meet everybody's needs well.