Grades are reliable to one grade either way

[gradeunreliability]

I’m a student and have just read this new book, released last week (www.canburypress.com/products/missing-the-mark-dennis-sherwood-isbn-9781912454990), shows how as many as 1.5 million grades are wrong each year, across GCSE, AS and A-Level subjects. That’s 25%! I have read the book and it’s opened my eyes massively. Grades can be one - sometimes, even two - grades adrift. I really encourage all students and parents to read this, as this can’t be acceptable if it affects university courses or sixth form placements?

This is a big reason I favour a post results application system, OP - speaking as someone who did a long stint as an Admissions Tutor in a large School at a Russell Group university. Evidence suggests both that certain schools over predict and that some schools that don’t send many students to uni under predict. Then of course there is the fact that even the best predictions are fallible.

It would require a major rethink and the cooperation of all stakeholders, but fellow Admissions Tutors and I have ironed out a number of the wrinkles. And the extra effort would be offset by great savings in other efforts for all stakeholders. There are a couple of wrinkles in specialist disciplines that we ( who are in STEM) haven’t fully grappled with, but for the most part it is a question of priorities as we think the Year 1 academic calendar would need to be rejigged. That is a hard sell at research universities.

This is such a great idea. I know Liz Truss suggested possibly moving the academic year to January for Year 1 university but that was to give all A* students Oxbridge interviews. The trouble is that if grades are unreliable, I suppose she has to give this to all students who receive A’s and even B’s, since the evidence indicates grades can be two adrift too.

The process does require a huge overhaul and I feel the stakeholders need to really consider those their impacting - the children - who, for many, are negatively impacted.

I favour a process where students apply to university with their results in hand. The current system allows some to chance their luck.
It's crazy that some applicants are unable to be considered for a course even though on results day other students may be awarded a place with lower grades.

The massive rise in unconditional results makes it even more unfair.

@poetryandwine The book @gradeunreliability was referring to isn't about predicted grades. It's about actual grades. About 25% of them are incorrect, i.e. if the paper was marked by four different markers, one of them would give a different grade to the other three, so if that's the marker who's marking your paper, you get the different grade. This percentage is much lower for maths subjects and higher for humanities subjects.

There have been two full consultations on post-qualification applications in my time working in admissions. The latest one just ended earlier this year with Nadhim Zahawai (then secretary of state for education) deciding not to proceed with it.

This is deeply depressing. Let’s hope a more competent Education Secretary soon reaches power.

I've just come across this, which is pretty scary too.. www.schoolmanagementplus.com/heads-governors-school-leadership-governance/exams-one-wrong-grade-is-a-potential-tragedy-for-university-applicants/

Generally speaking, exam results for hard sciences are pretty consistent, in that the same piece of work will receive the a very similar mark if graded by different markers; but as the topic becomes more subjective (e.g. creative writing) marks given by different examiners for the same piece of work can diverge. That's something that is present in all exam systems and is not news. It is also why an appeals / re-mark procedure exists.

I also favour a post-results application system.
Practically, this would mean exam boards would need to publish A level results a couple of weeks earlier, there would need to be a window for appeals/re-marks in early August, with a deadline for Ucas applications in mid-late August. All uni admissions would then work much like clearing works now, but with the huge clarification that only grades already achieved would count. Unis would need to make admissions in a compressed time window from late August to end September, but they would be doing so with the actual grades and UMS scores in hand and would for the most part only be dealing with candidates who have met the standard offer for the course. This would probably translate into a system with more emphasis on exam performance and less on interviews.

Or rather, that's why an appeals/re-mark system did exist. In 2016, Ofqual changed the rules for appeals so that you could get a re-mark only if you can prove there has been a "marking error". So the fact that different examiners can give different marks was explicitly excluded as being "an unfair second bite of the cherry" - www.gov.uk/government/news/fairness-at-the-heart-of-proposed-changes-to-marking-reviews-and-appeals-system.

That divergence of marks now results in a divergence of grades - implying that the grade actually awarded is the result of who happened to mark your script. For example, if your mark is at, or close to, a grade boundary, it's a 50:50 chance which grade you get. Even in Maths. Without the right of appeal.

This is entirely possible given that where my DD has 2 teachers for the same A level subject one says she is an A student the other says a C student. So this is why we have wildly varying results.

In my subject (English) we experience this a great deal. In one GCSE paper, for example, there is an 8 mark tolerance across the paper (as it is high value). So that means, one student could be marked by one examiner as achieving 30 marks, but the same paper could be marked by a different examiner and achieve 36 marks and either mark would be acceptable as it is within tolerance and even after appeal would not be changed. This could be the difference of a grade.

I also favour a post-results application system.
Practically, this would mean exam boards would need to publish A level results a couple of weeks earlier, there would need to be a window for appeals/re-marks in early August, with a deadline for Ucas applications in mid-late August.

That wouldn't leave anywhere near enough time for courses that select by interview or audition. This is one of the major issues that has led to a post-qualification applications being rejected following both of the national consultations.

Also, nobody has been able to suggest a workable solution to provision of advice to applicants if the application window is during the school holidays.

This is so disappointing: the system surely needs some more clarity.

The book really does cover this issue and I’m going to try and circulate it because this surely can’t be allowed from such a fundamental public service? If more students knew about this, there would be so much uproar.

I did some more research and saw that the Times Education Commission did a report on this, using ‘Rethinking Assessment’ analysis. This information is in the public domain, why don’t Ofqual fix it?
nuk-tnl-editorial-prod-staticassets.s3.amazonaws.com/2022/education-commission/Times%20Education%20Commission%20final%20report.pdf.

All those in power for the last couple of decades have resented change. The application process is integral but if we can’t get that right with reliable grades, there’s very little hope.

The system needs a reform - personally, I believe the university year needs to be delayed and made up with shorter breaks.

People might be interested in this article published today on the Higher Education Policy Institute web site:

www.hepi.ac.uk/2022/08/16/grade-expectations

I haven't read any of the links, but stuff like English is subjective. I don't think you will ever get a system where there is 100% consistency. Anyway 1 examiner doesn't mark whole papers do they?

Lots of essay-style subjects are subjective but it’s not fair for students to be penalised because of a faulty exam system. We have the capabilities of setting up an online system for marking, which Ofqual are yet to deliver. We have the possibility of tightening up mark schemes, which Ofqual are yet to do. They have become so driven to reduce grade inflation, that they fail to address the fact that 1 in 4 grades are wrong. No system will be flawless but there surely has to be a way to reduce the unreliability. Until these changes are implemented, they should print that they recognise grades are “reliable to one grade either way”.

I may be suffering from a failure of imagination, but to me, any essay based subject is bound to be subject to variation of marking. Given that grade boundaries are always separated by only 1 mark, there are always going to be lucky and unlucky ones. The best way to avoid being unlucky and dropping down a grade is to do even better in the exam.

You’re right that an essay subject will always have some variation but the concern is for that student that needs an A at her A-Level English Literature but her examiner has given her one mark less than an A, resulting in an A. The student misses out on their university place. However, the next marker might have awarded the script one mark more, meaning they get into their desired university. The grades can change life paths and it’s so crucial to get it right. So many students are unaware of this too. There is a defunct appeals system, so it’s not like this can be checked on or assisted by a re-mark. That is why the system is so unfair - the marking is done with the best intentions but the variability can severely impact lives.

Sorry the student is given one mark less than an A and therefore, is awarded a B grade.

6 years ago with DS1 we had a tough year and he had dyslexia and no real support from us at home due to a relative being in hospital for many months so the focus was not on him and lots of caring for younger siblings. He soldiered on and on results day got 2 exam results which were good but not stellar his uni gave him a place however his third was a fail. It was a hard science and we knew without some more support it was going to be a D or E (supported with additional tutoring from us would have been a C). When we saw the raw marks it was only a fail by 1 mark. So we asked for a remark - as it was not essential for the uni place we had to pay. I didn't know which paper to choose so I paid to remark all three. I think £120 alot of money on a punt. The office staff thought I was nuts but When the remark came back ALL three papers raw marks had moved - some up and some down but the net position was a single mark extra and a pass!! So we got the money back and my son got his A level which he is so proud of because it reminds him of a time when life was super tough and he achieved. On Thursday I will check the raw scores and request a remark if very close to the next grade.

Fruitygal - that's a cheering story. A few years ago, Ofqual changed the rules for appeals so you can no longer "ask for a re-mark". To get a re-mark now you have to show that the original mark was subject to a "marking error", such as a failure for an examiner to comply with the mark scheme. The fact that two different examiners might - and indeed do - give the same script different marks is no longer allowed.

@fairgrader this is what happened they added up the marks again and they had made errors with all papers in the set of 3 - he has a Masters now and a great job in his chosen field x

Ah! Thank you! That is indeed a very bad administrative error. Yes Ofqual will correct those, when they're discovered. But the primary reason why 200,000 A level grades will be wrong on Thursday, and 1.25 million GCSE grades wrong the following week is not because people make errors in addition. The reason is attributable to the fact that different examiners have different opinions, some being on one side of a grade boundary, some being on the other side. So the grade you end up with is the result not just of your efforts, but also of the lottery of which examiner happens to mark your script.

Examiners do not know what the grade boundaries are. The work with mark schemes but the levels in those do not correspond to grades. Grades are set at a much higher level, based on the performance of the paper.

This all highlights the well-known problem of cliff-edge grade boundaries.

Say a particular A-Level has grade boundaries of:
70+ = A*
62-69 = A
56-61 = B
55-60 = C
46-54 = D
31-45 = E

A student scoring 69% gets 48 UCAS tariff points.
A student scoring 62% gets 48 UCAS tariff points
A student scoring 61% gets 40 UCAS tariff points
A student scoring 56% gets 40 UCAS tariff points

So a difference of 1% can cut off a student's preferred options, while a difference of 7% can have no effect at all on the grade. When there is - inevitably - a small or moderate difference in marks awarded by different examiners, when the marks are near the grade boundaries this can have enormous implications for the student's future.

Obviously the examiners don't know where the boundaries are: they aren't making conscious decisions about whether a student gets an A or a B. It's simply the fact that results are banded into grades rather than reported on a continuous scale that causes major problems. Of course, putting marks from different exam boards' papers onto a common uniform mark scale, and making this comparable between subjects, is enormously - and probably infinitely - difficult.