Attainment/Progress

[stubiff]

Thought I’d put up a graph to hopefully highlight the difficulty in just using either an attainment measure or a progress measure.
Here, I’m not talking about whether Attainment8 and/or Progress8 are good, but the use of the them. Also, not talking about ONLY using these measures.

It is beneficial to use both the measures in tandem. You hear quite a few people say ‘School X has better results (attainment) than School Y’, but that isn’t the whole picture. Generally, GCSE attainment is directly related to prior attainment. The range of ‘progress’ between schools can be large.

Average A8 is 49.8, average P8 is -0.03, for State-funded schools.

Schools 1 and 2 have the same attainment but significantly different progress.
Schools 3 and 4 have the same progress but significantly different attainment.
Schools 5 and 6 are the choices I have. School 6 has a bit better attainment and significantly better progress. What the underlying data shows is that both schools have a similar cohort making them easy to compare, so School 6 is progressing pupils and School 5 is regressing them (even though the latter’s attainment is ok, the same as the national average).

The data is taken from the Gov website, 2016 Provisional.

1-4 are random schools, but which would you choose - School 4 (hi progress, lo attainment) or School 2 (lo progress, hi attainment).

Other flavours of the measures you may want to take into account (which should be on the site (or at least in the download data) when the full results are published) are:
Progress by gender – girls generally perform better than boys, so a school that closes that gap or reverses it may be good for a DS and not so for a DD (depending on the reason for the difference).
Progress by prior attainment – a school with a P8 score of 0.2, say, may have significantly different values for Low, Mid and High (prior) Attainers, e.g. -0.2, 0.3, 0.1 or 0.3, 0.2, -0.2.

[schoolphone]

Based on just the graph shown surely school 1 out of all 6. If just between school 2 and 4 then it would be 4. Unless you have more information on progress for different attainers and you know you child is clearly low/med/high. However most importantly you and your child have to actually like the school and its values
when you visit.

[stubiff]

schoolphone, 5 or 6 are my actual choices. 1-4 are there to show the potential difference in attainment and progress, in pairs.
So you favour progress (4 over 2), even though the progress of 2 is not that much below average.

Didn't really want to get into 'visiting/values/gut feel', etc, as that is a given, just a conversation about attainment and progress.

[schoolphone]

I would favour progress but have no idea on the confidence intervals or the scale of this progress shown. If the difference in progress is not statistically significant then I wouldn't consider it much at all. However personally I would only really consider progress based on low/med/high performing children rather than overall as averages don't tell you much. For example I would choose a school that makes excellent progress with high performing children and ideally sets them 1 to 9 from the start in all subjects if your child is very able and certain to be in top set
.

[stubiff]

schoolphone, can't disagree with any of that.

[DataEducator]

You have to be careful with the progress measure at the moment. It still remains - at least partially, a measure of curriculum choice.

So if a school runs a three year KS4 it is penalised slightly in this first year, as those students would have chosen options in Spring 2013, at the exact same time as the measure was first announced.

Those schools with students choosing options the year after would have been at an advantage to direct more students into Ebacc subjects and therefore generate more points towards the progress measure.

Because progress 8 is a zero sum game, i.e. winners in the system have to be offset by losers, it will take a couple of years for the measure to become more reliable.

I know it's of no use at the moment, but in January, you will be able to see the progress performance in English, maths, ebacc and open group subjects. This will give you a better picture. i.e. both schools 5 and 6 might score the same in english, maths and open, but school 5 did not have a 'good' curriculum model for the p8 measure at the time.

[stubiff]

DataEducator, understood.

The two schools have Eng Lit and Dual Science as compulsory so they only need to bullycoerce the students into one more Ebacc subject.

Is Eng Lit and Dual Science compulsory at GCSE in all schools now. I know Lit wasn't.

School 6 has their 2015 P8 on the site and has a reasonable 'spread' of values across the pots. Am guessing School 5 didn't put theirs on as it was so poor (based on 2016 value)!

[titchy]

Dataeducator can you explain how p8 is a zero sum game. Isn't it theoretically possible for all schools (pretend this is Gove utopia...) to improve all pupils' expected GCSE average grade from their KS2 position? In which case all schools would have a positive p8 score.

[stubiff]

titchy, all schools could progress beyond GCSE estimates, however, P8 is calculated based on the results of similar cohorts.
Let's say there are 100 schools with exactly the same cohort, i.e. the same average KS2 scores. Regardless of the absolute progress, the middle section will have a P8 of 0.0 (i.e. the schools who made average progress within that set of 100).

It's roughly like making 50% of the pupils fail an exam, regardless of their ability. They may all get in the 90's but the low 90's will fail (i.e. progress < 0.0).

[HarveySchlumpfenburger]

It's only like making half of students fail if the 'pass' rate is 0. But it isn't, it's a negative number. All of the schools on your graph are above the floor standard of -0.5.

[stubiff]

Rafals, agreed, the gov pass rate is -0.5, but I think parents would be looking at < 0 as a 'fail', I.e. below average.
A floor standard of -0.5 means the pass rate is very low. Last year only 300 odd schools out of thousands were below the floor.
I'd want to be sending my child to a school which is above average (>0) not, just, slightly above the floor.

[stubiff]

Low Pass mark rather than low pass rate.

[HarveySchlumpfenburger]

Which is one of the reasons progress 8 is problematic. Parents using statistics they don't understand to make decisions is an issue.

Even if the progress 8 scores didn't unfairly disadvantage schools with lower attaining cohorts on entry, the difference between 0 and -0.2 is negligible in real terms.

[DataEducator]

Well -0.2 in plainer language is a fifth of a grade less progress across 8 subjects than the average of similar students nationally.

However, back to some other points:

Titchy - as progress 8 is an after the event measure, i.e. it reflects what happened not what was predicted to happen it means that for every student above the 0, another must be below.

In regards to the original post, as I said, the measure is still in it's infancy so I would check the history of school 5 and 6 in terms of value added (similar measure to P8).

Also do not underestimate the 'power of one' in progress 8, every student, every grade counts. However one student with high prior attainment who fails to sit exams through no fault of the school (say unexpected illness), really impacts the measure. Much more than a threshold measure. I think the government should remove outliers from the data and publish the measure based on 90-95% of the cohort as really what we want to understand is the typicality of the progress made, not see figures skewed by unusual circumstances.

Finally, remember five years is a long time in education.

[titchy]

Thanks data - I'd missed that p8 is a relative score - didn't realise it wasn't an absolute score. That's not exactly clear to parents. Pity Gove isn't in charge though - he'd be requiring all schools to have a positive score!

[stubiff]

Rafals, re 'Which is one of the reasons progress 8 is problematic. Parents using statistics they don't understand to make decisions is an issue.'
Which part of my post are you referring to. Are you suggesting I don't understand or parents, in general, don't understand?
Thanks.

[stubiff]

I tend to agree with DataEducator. Taking my real life example, the net difference between schools 5 and 6 is 0.71. For me that is significant, it could be the difference between getting 4 or 5 A*-C's, or getting the grade needed to study a subject at A-level.

The previous Value Add of the schools is of the same order as the P8.

P8 may be problematic, but it is easier to understand than Value Add in respect of the 'value' is expressed as a proportion of a GCSE grade.
A Value Add of 1015 has no scale, aside from it's above average, equiv to a P8 > 0.

[HarveySchlumpfenburger]

Parents in general.

I think that parents will look at that negative sign and think that any school with a negative P8 score has poor progress results. Statistically that isn't true. Schools that are 'average' will have a range of P8 scores that fall across 0. That's the nature of the measure.