45% of students set to receive an 'unreliable' English GCSE grade on Thursday

[noblegiraffe]

Well this is a worrying story.
Basically, due to marking inconsistencies, especially in a subjective subject like English, grades which fall close to a grade boundary are unreliable, because a different marker could have legitimately marked it differently.

The majority of students used to receive a grade A*-D, so 4 grade boundaries. Now that will be from 9-3, so 6 grade boundaries. As the number of grade boundaries increases, the number of students who fall close to a grade boundary and thus have an unreliable grade increases. This represents an increase from 30% of grades being unreliable to 45%.

The article suggests that grades should be scrapped, as the difference between a 79 and an 80 is slight, but one mark could get a B and the other an A, which will be viewed differently.

www.telegraph.co.uk/education/2017/08/20/thousands-students-set-receive-wrong-gcse-mark-new-system-experts/?utm_source=dlvr.it&utm_medium=twitter

Obviously the same problem will affect maths, but the marking is usually more objective.

Piggywaspushed is right: the 'blackmail' is real, as is the cost of appeals - both are disincentives to seeking a remark. Even worse is the fact that the appeals system is based on the principle that the 'victim' has to shout 'this hurts', and if the 'victim' does not shout, the mistake remains uncorrected. The 'victim', though, does not know that a mistake has in fact been made: most people trust the 'system', and are more likely to think "Oh dear, I didn't do as well as I had hoped" rather than "the markers have made a mistake". So many errors go uncorrected simply because the 'victims' do not appeal.

Fundamentally, the marking and grading systems should be such that all grades are first-time-right, and that, if an appeal is made, the likelihood should be that the originally-awarded grade is confirmed, rather than there being a 1 in 6 chance that an up-grade is awarded (that figure of 1 in 6 is as published by Ofqual's annual statistics - see Table 2 of www.gov.uk/government/uploads/system/uploads/attachment_data/file/577360/Reviews_of_marking_and_moderation_for_GCSE_and_GCE_summer_2016_.pdf: in 2016 there were 280,250 appeals for GCSE resulting in 51,150 upgrades).

And for a statistically better way, take a look at the blogs on www.silverbulletmachine.com!

Have you got any idea what the stats are on a downgradeSherwood? ( I ought to check your attachment before asking but you seem so wise!)

And don't get me started on the financial disincentives!! State schools simply cannot afford all this - and nor can the families who the costs are being passed on to in many many cases...

For some reason DS's school hasn't given the breakdown of individual paper scores for the AQA English exams taken - they've just got the 'Levels' achieved. Seems a bit strange when for all other exams taken he's got the breakdown of marks across all the papers. Why would that be?

I am inclined, given the info/concerns on here about AQA English marking, to consider remarks (particularly as DS was predicted higher for Eng. Literature than he achieved).

Haven't got loads of money to be throwing at remarks across the board but would do so if it is more likely that the marks could be incorrect which could possibly be the case for AQA Eng. Lang/Lit. (from what's being said on here)?

Are there any repeat patterns in the remark 'upgrades' across the different subject areas? So it's more likely that one could achieve one in something like Eng. Lit. than in Physics? Are there any available stats to support this?

My results are bizarre. As are the whole school. We do wjec/educas.
I had a set 2, most kids predicted 5-7 (but obviously aspirational grades as their target!). The range was actually 3-7. I just don't get it. These kids could not have worked harder and last year I would have said all but 1 or 2 would have got B/A grades. As far as we could tell they got good mock grades but these low scores are out of nowhere.

In response to Piggywaspushed:

According to the official Ofqual Stats, for the summer 2016 GCSEs, there were 280,250 appeals, resulting in a total of 51,350 re-grades: 51,150 up-grades and 200 down-grades. The (significantly) greater number of up-grades is due to the fact that most appeals are made for scripts marked just below a grade boundary. So why are there any down-grades at all? Because, for some subjects, the variability in marking can straddle two grade boundaries – so an appeal for a mark just below the upper grade boundary results in a re-mark just below the lower one, which is really unlucky, but can happen.

In general, the probability of an up-grade on appeal depends on three factors: (1) the mark being appealed, and its proximity to the grade boundary; (2) the subject, with, say, physics having a much narrower spread of marks as compared to, say, art history; and (3) the grade width – if the grade width is narrow, there is a risk of a down-grade.

Ofqual’s report “Marking Consistency Metrics” (www.gov.uk/government/publications/marking-consistency-metrics) contains some “interesting”, if not alarming, data on the probability of being awarded the right grade if the script is marked close to a grade boundary. The first diagram is for GCSE physics, where the probability of getting the right grade at a grade boundary is about 50:50 (you might as well toss a coin); the second diagram is for components within “an unnamed humanities subject”, where the probability of being awarded the right grade is about 50% at best, with about 25% (!!! yes 25% !!!) quite likely.

And in response to Teddygirlonce, the third diagram, also from “Marking Consistency Metrics”, contains some important information on English Literature. The diagram is very cluttered, but focus on the little white spots. You'll see that, for English Literature, the spot is at about 55%. That means that, in the past, the probability of being awarded the right grade for Eng Lit has been about 55% (with a lower probability for marks close to a grade boundary). So the probability of being awarded a wrong grade is about 45% (and higher close to a grade boundary).

But that was when the grades were A*, A, B... This year, the grades are 9, 8, 7... and the grade widths are narrower. So the probability of being awarded a wrong grade this year is more likely to be around 45% x 6/4 = 67.5%!!! That seems to me to be "reasonable" grounds for appeal!

The current grading system is a mess. It’s unfair. And what’s worse is that so many people ‘trust the system’. It must be fixed, and Ofqual need to be put under pressure to do so. So please spread the word!

Really fascinating. And troubling!

Can't think what would work , though... is it possible to have a fair and unflawed system?

I had one pupil who got a 9 on Lit and 5 on Lang. I predicted 8 and 7, looking at his marks per question I discovered he had been given 0 for Q5 P1. AQA sorted it instantly and the additional 36 marks got him an 8. However I do wonder if his script was one of the 70000 still being marked in early August hence the admin failure. I would expect quite a few similar mistakes and, since e-AQA keeps crashing due to traffic, I suspect there are a lot of people analysing results almost forensically.

Yes, there is a better way - a way that ensures that almost no candidate receives a grade lower than he or she deserves. Essentially no candidates are disadvantaged, and no candidates suffer (often unknowingly) the loss of important life chances.

To do this, we need to measure the variability in marking - the spread of marks given by a panel of markers to a single script. For any examination, this can be done, quite easily and statistically correctly, using a sample of scripts. Call this number 'f' (say 2 marks).

Because judging an essay is not "an exact science", any script given a (single) mark 'm' (say 64) by a (single) marker is very likely to be given a different mark by a different marker - but if 'f' is measured statistically correctly, that different mark is very unlikely to be less than 62 (= 64 - 2 = m - f ) or greater than 66 (= 64 + 2 = m + f).

Currently, the grade is determined by the mark m (=64). If the grade boundary is 65, the grade awarded is, say, a C. Suppose that an appeal is made. When that script is re-marked, it is almost certain that the re-mark will be in the range from 62 to 66. If the script is re-marked 66, then the grade will be changed to a B. But It could also be re-marked 63, and so the grade C is confirmed - that's why not all appeals result in an up-grade.

Because the range of legitimate marks 64 ± 2 =m ± f straddles the grade boundary, is it a matter of luck which side of the grade boundary the first marker's mark happens to be. And it is the first marker's mark that determines the grade - hence the current mess.

So, a solution. When assigning the grade, do this not on the first marker's mark 64 (= m), but give the candidate the "benefit of the doubt" and assign the grade according to the 'adjusted' mark 66 (= m + f ).

If the same quantity f (in this case 2) is added to all candidates' marks m, then all candidates are treated fairly. And, as explained in the blogs on www.silverbulletmachine.com, this doesn't drive grade inflation!

The value of f, the measurement of the (legitimate) variability in marking, is likely to be different for different subjects: for maths and physics, the number will be relatively small; for art and English Literature, relatively large. That's why the lines in the Ofqual diagram are where they are, with the physics line "higher" than the history one.

That's kind of what we used to do when marking coursework, in reality : and now that is gone of course. So much politics!

Reds looking at your post I should be able to get a mark per question out of my DS's school shouldn't I??

Given how much effort goes into moderating grades given by the school for coursework or practical elements of an subject. surely it should be possible to apply a similar moderation scheme to the examiners to reduce the subjectivity in marking humanities subjects?

A few days ago Piggywaspushed wrote

"And don't get me started on the financial disincentives!! State schools simply cannot afford all this - and nor can the families who the costs are being passed on to in many many cases..."

It is absolutely true that the fee is a huge disincentive to appeal. What is even worse is the recently introduced 'whole centre' policy, whereby a school can be obliged to have all pupils' scripts re-marked, so putting the school in the position of risking that an appeal for a script marked just below a grade boundary, resulting in an up-grade, might cause a script marked just above a grade boundary to be down-graded. This is pernicious, and grotesquely unfair.

As regards fees, is there a lawyer in the house? It so happened that on 26 July, the Supreme Court ruled that the government was acting unlawfully when it introduced a fee for bringing a case before an employment tribunal, on the grounds that the fee acts as a barrier to justice, and as a disincentive to a victim to lodge a complaint (see, for example, www.bbc.com/news/uk-40727400). Mmm… I wonder if the same could be said of the fees for exam appeals, especially this year, given all the disruption attributable to all the recent changes...

Good points!

Exam boars running themselves as cut throat businesses is a truly dreadful thing.

BOARDS - but I prefer my Freudian Slip...

Dennishsherwood - good post, but one thing you've missed is that grade boundaries in the 9 to 1 subjects have been made wider, not narrower and it has been done deliberately to avoid the problem being discussed on this thread.

Mmm... I'm puzzled. I thought the policy was that the new 4/3 boundary is, by definition, the same as the old C/D boundary. Since the top mark is still ( say) 100, the six new grades 9, 8, 7, 6, 5, and 4 therefore cover the same space as the four old grades A, A, B and C. It seems to me that the average grade width must be narrower. It is possible, of course, that on comparing this year's 9, 8, 7... boundaries with last year's A, A, B... boundaries, the 9, 8, 7... grade widths are wider, but the only way for this to happen is for this year's 4/3 boundary to be way lower than last year's C/D boundary - as has probably happened with this year's GCSE maths. I suggest, however, that the last year/this year comparison, although interesting, is not as important as the comparison between what has actually happened this year for 9, 8, 7... grades and what would have happened had this year's cohort been graded A, A, B... By this comparison, for English Language GCSE, "only" 30% of candidates would have been awarded the wrong grade had they been graded A, A, B... (that's quoting Ofqual's own data, see Figure 14 in "Marking Consistency Metrics" www.gov.uk/government/publications/marking-consistency-metrics). What has actually happened this year, under 9, 8, 7... grading, is that I think up to 45% of candidates have been awarded the wrong grade - or rather, if the entire cohort were to appeal, and to be fairly re-marked, 55% of candidates would have their grades confirmed, and 45% would be changed - about 22.5% and 22.5% down.

The 45% wrong grade number was first published in an article in the Sunday Telegraph on 20th August. On 22nd August, Ofqual published a press release www.gov.uk/government/news/response-to-daily-telegraph-story-on-new-gcse-grades. This does not refute this number. I wonder why not...

And tiggytape, you are right, thank you: consent is required. Which is why, understandably, Heads are very reluctant to appeal on behalf of a whole cohort - they can't risk the down-grading of some of the students for the sake of the up-grading of others. So the Head has to make individual appeals on behalf of individual candidates - and cough up the fee accordingly.

That release clearly says there is a 'better spread of grade boundaries'

....

"New GCSEs have been designed from first principles to deliver better differentiation on the new 9 to 1 grading scale. The new GCSE exams and mark schemes have been created to support the increase in the number of grades, with better spread of grade boundaries and reliable assessment.”

Whatever a 'better spread' might mean; and it would be nice if they were to measure 'reliable'...

What's important in this press release, is that Ofqual make no statement about the allegation that 45% of grades in this year's GCSE might be wrong - an allegation that has now been reported in many sources (including mums net!!!). That press release is dated 22 August, today is 30 August, and still no statement from Ofqual. I would have thought that, had that number been significantly - or only modestly - wrong, Ofqual would have published a rebuttal. They haven't. Why not?

Let me make those numbers real. Each year, about 500,000 candidates take GCSE English Language. If graded A, A, B... about 30% = 150,000 candidates this year would have received a grade which would be changed in appeal, about 75,000 resulting in a down-grade, and 75,000 in an up-grade. But since this year's cohort has been graded 9, 8, 7..., I estimate that up to 45% = 225,000 candidates have received the wrong grade. That's 75,000 - yes, 75,000 - more than had this year's grading been A, A, B... And maybe one of those candidates with the wrong grade is your DC! But how would you ever know?

For English Literature, with about 400,000 candidates, the number of grading errors this year could be as high as 45% x 6/4 = 67.5%. That's way more than half the cohort, corresponding to about 270,000 candidates. And the probability of a grading error for scripts marked close to a grade boundary is even higher.

@dennishsherwood that is shocking...

Yes, it is indeed. And the more people that know about it the better. So let's use our networks and social media to spread the word! Very importantly, if your DC has taken GCSE or A level in any subject this year, even if the awards were 9s and A*s, speak to your school too. We need to put public pressure on Ofqual to fix it - and the fix is very easy to do (as outlined in m post at about 2 pm on 28 August).

(Sorry about that erroneous comma between 'release' and 'is' in my previous post - separating a subject from its verb by a comma would be a GCSE fail!)

Just wondering how long it takes a school to get a breakdown of marks from AQA? I asked for English Language and Literature 2 weeks ago but haven't heard anything yet.

it should not take any time at all. My guess is , because you asked in the holidays, it was overlooked. I'd ask again. Who did you ask? I'd go to the head of subject as my DS's exams officer didn't give me the full breakdown.

Today's Telegraph contains another article on the unreliability of GCSE grades www.telegraph.co.uk/education/2017/10/03/children-having-careers-blighted-due-inaccurate-exam-grades/.

To make that real, there is very good evidence that for this summer's GCSE and A levels, ONE grade in every EIGHT would have received an upgrade had an appeal been made. That means that it is quite likely that every student who sat eight or more GCSEs was awarded at least one grade that would have been upgraded on appeal. Yes, every student. Which might include your DC. So this issue is really important - all the moreso because over 90% of those who are 'eligible' for an upgrade don't appeal - they just take it on the chin.

The Telegraph article is based on a speech given earlier this week by Chris King, the Chair of the Headmasters' and Headmistresses' Conference (HMC), the 'club' representing fee-paying schools. In the full text of his speech www.hmc.org.uk/blog/2017-hmc-annual-conference-media-briefing-chairs-annual-address-3/, you will see these words

"Mr King will announce a new joint school leaders’ policy summit, hosted by ASCL, to consider solutions to the large number of unreliable exam grades awarded to students. The exam regulator’s own figures estimate that more than a third of candidates do not receive an accurate grade."

ASCL, www.ascl.org.uk, is one of the 'clubs' representing the heads of state-funded schools and colleges, with whom HMC are collaborating to run a "policy summit" to explore all this. This is a very useful step forward, helping to build a consensus across all schools that will put pressure on Ofqual to fix things.

The more people that know about all this - especially parents and teachers - the better, so that heads can be given the support that will be so valuable in forging consensus at this "summit". So please network all this around!

There is a lot more information on "The Great Grading Scandal" on the blog page of my website www.silverbulletmachine.com, and for this who might be interested - especially teachers - I have available a spreadsheet "grade (un)reliability ready reckoner", which shows, very vividly, how unreliable grades are, and how sensitive grade reliability is to grade boundaries. If this is of interest, please contact me and I will send you a copy.

And Michelle Meadows, Ofqual's Executive Director of Strategy, Risk and Research, needs to listen harder. In today's Telegraph article, she is quoted as saying "Our [Ofqual's] extensive and ongoing research does not support HMC's statement about marking accuracy". HMC have made no statement about marking accuracy: HMC's statements are all about grading accuracy. Marking and grading are not the same thing, as Michelle Meadows surely knows.