"I'm not sure why privileged background = clever Statesman?"
It doesn't, always, but you'd have to be staggeringly naive to ignore the very very strong correlation between privilege and academic success.
Right from the word Go, kids from more privileged backgrounds are (in general) talked to and read to more, they are bathed in language, they spend far more of their leisure hours in environments which enrich their general knowledge and vocabulary (from aquariums to stately homes), they are taken to the library and have books bought for them. Their parents are usually stricter about time spent on TV and computer games.
As they get to school age, their parents hear them read more often, talk to them about things around them and things on the news more, and are more likely to incorporate maths practice into everyday tasks.
As they approach late primary and secondary, their parents often fork out for tutoring of various kinds.
All these factors have an enormous impact on educational attainment. Schools with privileged intakes should be getting a high rate of students to C or higher at GCSE without even trying very hard--these are not terribly difficult exams, as I said.