Why?
Suppose, for the sake of argument and this is completely hypothetical but could easily be true, your cleaner had the same surname as you, quite a usual western name, Campbell or Johnson, let's say, She -almost certainly a she - comes from the Caribbean. You know that your ancestors had estates in the Caribbean which they may or may not have run well. They also operated trading fleets out of Liverpool or Glasgow . You know that she is well below the poverty line, is just about existing on benefits and a low paid job. She's a single parent, relatively low levels of education, lives in a poor area where the schools have improved but the statistical probability of her children going on to tertiary education is lower than the national average . You discover, somehow, through the register and her family traditions, that her ancestors were slaves on your ancestors' estates. You don't know whether they came out on your other ancestors' ships but there's a chance.
Your background means that your ancestors' money cane in from the estates and also the ships and that since the early 19c all the men who were capable of picking up a spoon have gone to university and since the early 20c most of the women too.
Don't you think you might feel just a slightly complex emotion about the disparity and the causes? Perhaps a tiny bit of shame or guilt and/or some enhanced level of responsibility/ determination to redress the balance? Or that you could imagine or understand doing so?
I agree it could be - not is - self indulgent if you don't do anything about it. You'll note that I try to.
I'm also not saying everyone should feel guilty. But it's understandable and not necessarily self indulgent.
But in any case, as I said, emotions and logic are not always hand in hand, as you may discover.