More that 1.5 million wrong grades will be awarded at A level, AS and GCSE this year

[dennishsherwood]

Across all subjects, and across A level, AS and GCSE, about 1 grade in every 4, as awarded this August, is WRONG. That amounts to over 1.5 million wrong grades this year. But no-one knows which specific grades, or to which specific candidates. So maybe your DD or DS is an unwitting victim.

In response to a hum-dinger of an article on the front page of the Sunday Times on 11 August (www.thetimes.co.uk/edition/news/revealed-a-level-results-are-48-wrong-xsj33jvnh), Ofqual immediately posted a statement on their website (www.gov.uk/government/news/response-to-sunday-times-story-about-a-level-grades) which includes these words

"more than one grade could well be a legitimate reflection of a student’s performance ".

You might like to read that again, and think about it.

I think that what they are saying is "“grade [X], in subject [Y], is just one possible legitimate reflection of your performance, but there are others too - though we’re not telling you what those other grades might be, or whether they are higher or lower”.

To me, this is outrageous. And if you think so too, please complete the very brief survey on //www.surveymonkey.co.uk/r/gradereliability.

If you would like more information, and to check the evidence, click on //www.hepi.ac.uk/2019/08/15/dear-ofqual-%EF%BB%BF/ and follow the embedded links...

[Witchend]

So maybe your DD or DS is an unwitting victim. or has benefited from it-don't forget half the wrong ones will be too high, just less likely to be changed.

[dennishsherwood]

Yes, that's true. But I'd rather have an exam system for which all grades are reliable, rather than the current lottery of some right, some wrong upwards and some wrong downwards.

[Teachermaths]

They have to be marked by humans. Human error, differences in interpreting mark schemes and subjective essay based subjects all contribute.

How would you counteract this?

[PotteringAlong]

But the only way to eliminate any error and interpretation is to only have exams which have right or wrong answers. So facts only. Yet subjects like history rely on interpretation and that is the interpretation of both the candidate and the examiner.

How would you make it more reliable?

[dennishsherwood]

Agreed. Marking is inherently "fuzzy". There is no single "right" mark for a script (say, 54); rather, there is a range of marks (say, from 53 to 57), where this range captures the reality that different examiners can give different marks.

If grade C is all marks from 51 to 55, and grade B, all marks from 56 to 60, then if the script is marked 54, the award is grade C. But if the mark is 56, grade B. That's the problem.

The current policy - "mark each script once and award the grade based on that mark" - fails to take the reality of "fuzziness" into account.

The solution is to recognise the inevitable "fuzziness" of marking in the policy used for assigning the grade. There are several sensible ways to do this - for example, any of numbers 10 to 22 described on //www.hepi.ac.uk/2019/07/16/students-will-be-given-more-than-1-5-million-wrong-gcse-as-and-a-level-grades-this-summer-here-are-some-potential-solutions-which-do-you-prefer/

[Teachermaths]

I've read that blog before.... Its very interesting. However I can't see employers, universities etc being able to cope with marks given like this. How would that system work?

The difference between subjects has a lot of impact here. Maths is right or wrong and there is very little "fuzziness". History is very fuzzy.

The current system isn't perfect however you can appeal if you think you've been hard done by.

[SoonerthanIthought]

This is interesting! Was one of options 1 - 22 to have two markers, and a third marker if the two disagree?

Is there also a case for getting rid of the 'new' (ok 9 yrs old) A level A grade - if the grades are as 'fuzzy' as this suggests, maybe it doesn't make sense to have more rather than less differentiation? I know they were introduced because some universities said they needed the As in order to be able to get the most able students - but if 1 in 4 grades could be different, then are they really fulfilling that function? I appreciate that maths/sciences may be less fuzzy which does dilute that argument in relation to them.

Is there any evidence that As predict those who get firsts in all subjects? - if so, then I gracefully retract my suggestion! If the evidence is that A s predict those who get firsts in maths, physics, etc, then keep them for those subjects and not others.

Sorry I've gone off track a bit, but not really - the more 'doubt' there is about grades, maybe the less they should be differentiated? And the A has led to many univ offers being made which include one or two A s - which imv puts a lot of pressure on students. I would not have wanted one of my dc to have one of those as their 'firm' offer, because I think it is a) too much of a lottery (see above) and b) too much pressure at a stressful time.

[Teachermaths]

Sooner the main issue with multiple marking is cost.
I don't know the answers regarding a*s and firsts. It would be interesting to know.

[dennishsherwood]

Hi Teachermaths.

Yes, some subjects are fuzzier than others (see //www.hepi.ac.uk/2019/01/15/1-school-exam-grade-in-4-is-wrong-does-this-matter/), and the reliability of the awarded grade also depends on the script's mark (see //www.hepi.ac.uk/2019/02/25/1-school-exam-grade-in-4-is-wrong-thats-the-good-news/).

Your point about communication is key - if you're having a (long!) cup of tea or coffee sometime, take a look at pages 86 to 88 on docs.wixstatic.com/ugd/7c5491_12113dfe5f4f4a6fb7c4509e940e8ff1.pdf.

And appealing isn't so easy: in 2106, Ofqual changed the rules to make it harder to appeal - see //www.silverbulletmachine.com/single-post/2018/10/28/Biting-the-poisoned-cherry---why-the-process-for-school-exams-is-so-unfair.

[StrumpersPlunkett]

Genuine question.
Has this always been the case?
In the 1980’s I had predicted grades of bccd at a level. Got a’s And b’s for my coursework. Attended the examinations. Write answers to all questions and gained 4xu as a result.

I did retakes and all was well but I have always been confused how those results could be true!!

[dennishsherwood]

Hi Soonerthanthought

The greater the number of grades, the narrower the grade widths, and the more unreliable the grades. So when 9, 8, 7, 6, 5, 4 replaced A*, A, B, C for GCSE, the grades became more unreliable overnight. Clever, eh?

I don't know anything about the correlation between GCSE or A level grades and degrees - but if GCSE and A level grades are themselves unreliable (as they indeed are), the baseline against which any correlations might be measured is wobbly indeed.

And double marking adds surprisingly little extra information, as discussed in the section "The double marking fallacy" on pages 122 to 125 on docs.wixstatic.com/ugd/7c5491_12113dfe5f4f4a6fb7c4509e940e8ff1.pdf

[dennishsherwood]

Hi StrumpersPlunkett

There's a paper published by AQA in 2005 - cerp.aqa.org.uk/sites/default/files/pdf_upload/CERP_RP_MM_01052005.pdf

The top of page 70 reads

"However, to not routinely report the levels of unreliability associated with examinations leaves awarding bodies open to suspicion and criticism. For example, Satterly (1994) suggests that the dependability of scores and grades in many external forms of assessment will continue to be unknown to users and candidates because reporting low reliabilities and large margins of error attached to marks or grades would be a source of embarrassment to awarding bodies. Indeed it is unlikely that an awarding body would unilaterally begin reporting reliability estimates or that any individual awarding body would be willing to accept the burden of educating test users in the meanings of those reliability estimates."

One of the authors of this paper is Dr Michelle Meadows, now Executive Director for Strategy, Risk and Research... at Ofqual...

[SoonerthanIthought]

Reviews are quite expensive, aren't they? which may deter lower income schools and dparents.

Interesting that 'double-marking' adds little to the marking. So is the real point that there is no such thing as a 'right' mark (as indeed ofqual's comment suggests!). If so, it is odd to think that students are losing places on the basis of not getting a particular mark (and very upsetting for those students, obviously).

Or do the universities already allow for this, so build in a margin of error? (ie "We want people who are at least a 'true' grade B standard in History, so we will require an A - then if they get an A we know that they are at least B standard."? Obviously I realise this is unlikely to be an explicit thought process/rationale! But it might be the practical outcome. Obviously unfortunate for those who get an actual B, or a C that could/should be a B.)

[SoonerthanIthought]

"The greater the number of grades, the narrower the grade widths, and the more unreliable the grades. So when 9, 8, 7, 6, 5, 4 replaced A*, A, B, C for GCSE, the grades became more unreliable overnight. Clever, eh?"

Hmm, and I gather that universities do differentiate between 8s and 9s for the most competitive courses. I can't remember what the rationale was for the changes - were there some universities and employers saying "we can't tell how clever people are if you don't add a A grade"'? Was there any discussion at the time that there may not be a 'right' grade and of the difficulty of differentiating accurately to these margins in some subjects? (I do accept that 100% in maths may be more objectively measurable!)

[dennishsherwood]

There are many Ofqual documents that define 'the right grade' (and sometimes 'the true grade') as "the grade corresponding to the mark given by a senior examiner". That would be fine if "the senior examiner" were to mark all the scripts in any specific subject - but with about 700,000 GCSE English Language scripts to get through, that would take him or her quite a while to do. And if there is more than just one "senior examiner", what confidence is there that they would all give any one script exactly the same mark?

The reality is that different examiners can give the same script legitimately different marks, which - depending on where the grade boundaries are, and the subject's "fuzziness" - might result in different grades, one of which might happen to be the same as the grade that corresponds to the mark that would have been given by "the senior examiner", had "the senior examiner" marked the script in the first place!!!

It is this desperate muddle that has - at last - been recognised by Ofqual's ground-breaking statement of 11 August (as it the top of this thread) that more than one grade is "a legitimate reflection of a candidate's performance".

And no, universities do not already allow for this - they (and everyone else too) trust the grade as awarded as 'right'. The fact that this trust is (unknowingly) misplaced is why this issue is so important.

Also grade 9 was introduced (by Michael Gove) specifically to identify the very top GCSE students. Which it would, if only it were reliable...

[SoonerthanIthought]

This is so interesting Dennis! Maybe this is why some universities/courses are prepared to drop grades - perhaps they have been aware of this issue for a while (or not?) But not all drop grades - and what is so annoying/upsetting for students is to miss their preferred place because they get one grade lower, in one subject, than they were offered - is this really necessary if the grades are only one of several possible "legitimate reflection [s] etc etc"?

Had anyone asked Michael Gove to add in a 9 to the GCSE grades - would be interesting to know if the universities themselves wanted it.

This isn't just an issue at the top end of the scale, but I suppose one reason we have so many grades is that a feature of English HE is the huge emphasis (by schools, parents, univs themselves) on calibrating/ranking universities - so it 'matters' whether the university takes an A, A or B student. It matters both to the university (except maybe it doesn't matter so much after all because as it turns out, 'legitimate reflection' etc etc); and to the student. Given the combination of two factors - a) huge importance is attached by some to attending O/b or other very high ranking RG; and (b) people can miss their Oxbridge and 'very high ranking' RGs offers by one grade or , this is perhaps not ideal.

Of course exactly the same applies to students missing other offers at CCD rather than CCC and so on.

Has any university yet commented on the Ofqual statement? Would be interesting to know their views!

[dennishsherwood]

Hi again SoonerthanIthought... thank you, and your points are all important. Everything you say is valid - and may I also throw in the critical importance of the 3 - 4 GCSE boundary in maths and Eng lang, which is a life-changer to many young people. My simulation of what happens at this boundary for Eng Lang is a reliability of about 40ish% - that means that more grades are wrong than right, and that tossing a coin would be more fair.

Yes, I too am awaiting the response to Ofqual's bombshell. The more people that know about this the better...

[Piggywaspushed]

The new thing spreading around schools is comparative judgement : I wouldn't be surprised if this makes its way into exam marking. Teachers who do it swear by it. It (my understanding , this) rank orders scripts first (which apparently nearly always throws up more or less the same rank order) in a very clever computer aided fashion (when done in a the truly proper fashion all the scripts are scanned in) and then AFTERWARDS awards marks based on a distribution curve.

My examiner friend is sceptical but I think it sounds like a way forward.

[Rosieposy4]

Would need a massive overhaul of exam marking though piggy. We mark question by question, which gives you much better standardisation within each question, and allows senior/ decent markers to pick up the slack where other markers have failed to complete. It would still give you marker to marker variation and for any of the larger subjects you talking 200+ markers per board per paper at GCSE

[Piggywaspushed]

I have huge doubts about question by question marking and its efficacy must be honest.

Comparative judgement doesn't require more markers. Anyone I know he has used it swears by it.

I have heard whisperings of it being introduced in A Level history within the next few years.

[woman19]

How would you make it more reliable
Pay examiners the living wage?

Also.

Exam boards have been utterly desperate for examiners, particularly since 2010. I wonder how many practising teachers are still doing marking, with all the other pressure on them.

In fact, I wonder who is doing most of the marking.

[Piggywaspushed]

True!

But , even in a fully qualified English department, we never agree (and certainly not with the examiners' exemplars!).

One thing we nearly always agree on is the rank order, though : which is where the comparative judgment thing comes from.

[woman19]

we never agree (and certainly not with the examiners' exemplars
Because the exams and mark schemes are so badly written and constructed.

English GCSE did not used to be like this.

The current high rate of anomalies in the examination system lets kids and parents down terribly.

[SoonerthanIthought]

Maybe the truth is that there is no way of accurately producing 6 A level grades and 9 GCSE (not including the ungradeds) without getting some 'wrong' grades (which then won't all be corrected because of the expense and also the threat of going down a grade, if you apply for a remark - see the other thread!)

If that's right, why do we need so many different grades? Partly, it seems, to allow universities to differentiate between the very brilliant, quite brilliant, good etc so that the univs can all be higher or lower in the 'rankings'. Why do they need to do this - couldn't they all recruit a wider ranging cohort (which I think is, to some extent, what they do in countries where students largely go to their local university)?

If we want to keep elite universities (ie universities which only take the very brilliant (again, starting from first principles, why?) those universities could set their own entrance exams (and in many cases already do!)

[Rosieposy4]

Piggy, just say I mark 350 scripts and rank order them ( i have absolutely no issues/worries at all with question by question marking btw, and use it at school even when I don’t have to. It allows me to get thoroughly conversant with a mark scheme, not to have to hold too much in my head or scribbled notes. I can be clear about whether i allowed x or not etc etc)
Then the next marker rank orders theirs, so 200 markers each with their own rank orders from 1-350, then the next , so you have 200 piles of 350 in rank order.
Obviously next then they need to to rank ordered within much bigger sets, and that is where the huge amount of extra work comes in.

(Post edited by MNHQ)