I think it is more the case that generational poverty wears down individuals and communities. There is a huge division between the rich and the poor in this country. I believe it is one of the most economically unequal places in Europe. Unfortunately, this division is reflected in the faces and the prospects of people. I live in an area of significant economic deprivation, and the population are not ugly, they just look poor, tired, stressed and perhaps unhealthy.
'The Rich' don't come here, but even if they did bless us with their 'good genes', it would not take long for the environment and the poverty to grind down the blessed individuals.
There was a time, fortunately now largely forgotten, when arguments were put forward towards promoting good breeding among the rich and elite, and limiting the procreative choices of the poor. The 'Poor' were observed to have bad health, bad teeth, they were smaller, weaker, more prone to disability than the 'Rich', and thus presumed to come from a weaker stock than the more privileged. The danger is that once we introduce genes into the equation between wealth and greater attractiveness, we are a little closer to revisiting the nightmare of quasi-eugenics that marred decades of the nineteenth and twentieth centuries.
Yet, it would be possible to show the probability of attractiveness is not primarily genetic. We could take a baby from the most privileged area, and swap it with a baby from the least (and vice versa) and the impact of the children's environments would soon outweigh their genetic predispositions. No-one looks attractive with poor teeth, bad nutrition and constant anxiety about meeting their basic needs.