The league tables are out - let's talk Progress 8

[noblegiraffe]

The secondary league tables were published this week:

www.compare-school-performance.service.gov.uk/schools-by-type?step=default&table=schools&for=secondary

Some schools at the top have fab progress 8. Harris Battersea has done amazingly with 80% PP

I saw on Twitter this interesting breakdown of Progress 8 by type of student. Girls make more progress than boys, students with EAL make more progress than those who have English as a first language. Disadvantaged pupils make less progress, students with SEN even less.
When you look at the league tables, you then notice at the top several girls schools, and London schools with a high proportion of EAL.

Is progress 8 really a measure of progress, or a measure of school intake? Does a student being EAL outweigh them being disadvantaged?

And why are boys doing so much worse than girls?

Just answered my own question - yes, that happens all the time. Ordering the schools in my county for P8 for sub-groups has a school with a small cohort of the specific subgroup listed as 'Average' for a P8 of about 0.8, while one with lots of them and a P8 of around 0.6 is listed as 'Well above average'.

Nowt so odd as statistics...

can't yes, it's an issue with applying these sorts of statistics to small groups. The confidence intervals are so wide as to make nonsense of the whole thing. It's bad enough on a school level, but when looking at a small subgroup, e.g. Low attainers, it's worse.

The picture from the OP will be more reliable than for individual schools because the samples are much larger.

the Problem’s going to be parents thinking they understand it and making judgements about schools, isn’t it? In theory the system looks simple and easy to understand but it’s actually not because two schools with different scores could be similar.

noble are you aware of the cock up with the ks2 progress measure when it comes to children with EAL/SEN?

The DfE were asked why they didn't come up with a system that was less shit than Progress 8 and the response was that parents wouldn't understand it. So apparently parents are the key stakeholders in this. And now we have something that's misleading and crap and parents still don't really understand it.

No rafa, what's that about EAL/SEN?

can't have you actually looked at the confidence intervals for each subgroup? You can see them from the league tables if you click on the (?) underneath Progress 8 score. Then you could see how ridiculous some of them are!
The progress 8 for the smallest subgroup of my school has a confidence interval 1.4 wide. They could actually be anywhere between amazing and crap.

I've just found a confidence interval of 1.97....

Funnily enough, the score is thus 'average' because it pretty much can't fail to include 0 within the range....

1.97 They might as well say 'I dunno, what do you think?'

Someone, somewhere has signed off on this. And I'm betting it's not a statistician!

The rather scary thing is, a range of 1.97 isn't even particularly rare.

As the cut off point for not showing the results is 5 pupils, if you can find any subgroup with only 6 -7 in it (disadvantaged pupils in grammar schools are pretty fertile ground for huge confidence intervals, I have found so far), odds are that the confidence interval is well over 1.5, with 1.97 being really quite common.

Wow I'm now so glad that I didn't send Ds to our catchment school. Even the awful private school he went to for 2 years before moving to his current school was a better choice.

Our catchment school is in the worst 100. It actually does very well with kids who enter with less that a SATS level 4 but poorly with average attainers & terribly with high attainers those entering with level 5 & above. (The group Ds would have been in). Out of this group the progress 8 was -1.9 & only 50% of them got a Level 5 in maths & English.

There’s a blog somewhere that explains it in detail but:

The system is based on the assumption that children with the same starting point have the same trajectories. But children in the lowest attainment bands at ks1 consist of mainly 2 groups - children with SEN and children with EAL. these groups have very different trajectories but the calculation treats them as being the same. When their ks2 scores are averaged out children with EAL end up with huge positive progress scores and children with SEN huge negative scores (double figures in plenty of cases).

I’d assumed that it might even out by the time p8 was calculated because there would be fewer EAL children in the lower attaining groups, but from your OP that might not be the case.

So as a parent of e.g. a low prior attainer, do you go for the school with an 'Average' Progress 8 of 0.79 (average because the very lowest end of is confidence interval is 0.01, though the highest is over 1.6) or the 'Well above average' one of 0.69, with a confidence interval of c. 0.2 to 1.1???

Alexander, what's the confidence interval?

only 50% of them got a Level 5 in maths & English.

That's more useful to know than the progress score, tbh. High attainers should be passing their GCSEs, that's really appalling.

Rafa Yes, EAL students may perform poorly due to language issues, but then quickly pick up English and perform well at the end of KS2.

I think the reason that EAL students continue to outperform other students at secondary school is different - the groups that make the most progress (by far) are Chinese and Indian children (even those from disadvantaged backgrounds). Culture seems to be the key factor there.

And for a school that had 10 low attainers, 44 middle & 42 higher attainers it makes it even worse.

Yep, that's inexcusable.

EAL do well for various reasons but some of them are:

1. The often score less well in the KS2 English (reading) tests but they are on more accelerated trajectory that means once they nail the language barrier, they get good results relative to their depressed starting point.

1. They often take a GCSE in their other language(s), which is a fairly easy top grade.

1. Work ethic and culture concerning education and making the most of the opportunity.

P.S. glad you found my chart useful.

Progress 8 score and confidence interval

LOW 0.38 (-0.38. -1.5)
MIDDLE -0.54 (-0.91 to -0.17)
HIGH -1.09 (-1.45 to -0.73)

Only 9% of middle attainers got Grade 5.

Ah yes, @DataEducator, a really interesting chart, good job! I thought from the source that it was a government one. I saw it retweeted by Mr Drew from Educating Essex :)

Yes,there’s that too.

Although the combination of negative scores for children with SEN and positive score for EAL makes me think there might be something else going on.

Either way, I think it shows that you can’t just look at the figures without taking account of the cohort. 0.3 in a school with a large number of EAL is not really the same as 0.3 in a school with few EAL children but an above average number with SEN.

Is there anywhere that shows separate English and Maths results? I know our school did catastrophically at Maths- if would be good to see how they did in English.

You can get the P8 score for the subjects separately on the tables. For my school it is 0.11 for English and -0.02 for maths.

alexander have you looked to see what your local school got for grade 4+ (the ultimate hidden stat!). My school got 50% for En/ma at 5+ and 77 for 4+ !