My point was that if state schools have in the past managed to achieve better outcomes then what has changed?
Charley this is a reasonable question but I think it has a complex answer and not a simple one such a "leadership" (which implies discipline or authoritarian approach) and I think that if you ask 5 people, you will get 5 different answers.
I don't think that you can analyse the microcosm of the schools outside of the broader social and cultural landscape and I'm not very good at making the point but I will try.
In the past 30 years social mobility in the UK has declined, incomes have stagnated and vastly increasing social and economic inequality has emerged - work is no longer a clear route out of poverty and this has a very broad social impact, especially in schools. Huge government cuts have been regressive. 30 years ago a child had more of a chance of having a higher standard of living than their parents - many of today's children will have lower standard of living than their parents. In other words, social mobility is at risk of reversing in the UK.
In the US where the socio-economic divide is greater than it is here (and race plays a big part in the US), the "inner city" schools are violent, drug ridden and rates of incarceration, and recidivism are high. This is not a problem of education or lack of respect or leadership but is rather symptomatic of the social and economic climate at large. Children in the inner city schools have nearly zero chance of bettering their lives, they live to survive.
Also, 30 years ago in the UK there wasn't the rampant social/cultural shaming for being in social housing, for being on benefits, etc. The cult of the super-rich didn't exist, wealth and signifiers of wealth (cars, etc) weren't worshipped like they are now.
Here's an excerpt from an article about the vicious cycle of inner city schools in the USA and a snapshot of where the vast socio-economic divide can lead and how it can affect the classroom:
Students enter buildings through metal detectors. If the device goes off they are bodily searched. Armed police stand guard. Uniformed security crews that report to the police sweep the halls. Students are forced to sit in overcrowded uncomfortable classrooms doing rote assignments geared to high-stakes Common Core assessments. Stressed out teachers, fearful that they will be judged by poor student performance on these tests, use boredom and humiliation to maintain control of the classroom.
When young people react to these conditions they are disciplined. The ultimate goal of school policy is to sort them out with a few destined for success, some to menial jobs, and others for imprisonment. When these pipeline schools do their jobs well, the young people who fail are convinced they failed because it was their own fault and that they deserve their punishment.