to wonder how exam boards/ofqual can be so shit at statistics and if they even passed A Level maths themselves

[ShootsFruitsAndLeaves]

Roll a dice 30 times. How many sixes did you get?

www.random.org/dice/?num=30

I tried ten times:

5, 4, 6, 5, 6, 2, 7, 7, 4, 4.

Now imagine that's a comprehensive sixth form sitting an A Level. Even if we know the school's results for years and years, and we know 1 in 6 get As, then in any given year there are likely (99%) to be between 2 and 8 As in that class.

If you want to award the same number of As as last year, you're going to need to hand out 5 As to that school. But this year just by randomness maybe only 2 deserved it. Lucky kids. Or maybe 8 deserved it. Ouch.

Now imagine if that was a smaller class only 15, and normally on 1 in 15 get As. If student performance is randomly distributed then there's a 99% chance of 0, 1, 2 or 3 As. So you can have 3 A*s in a class that normally gets one. Not every school would be like that. But some classes in some schools WILL be in that position. And it's in fact NOT randomly distributed. Some schools will have good intake years. Maybe a genius physicist's twin children joined the school and are sitting A Level Physics. So the exam results are actually less likely to follow the past distribution than even random dice would be. It's even less predictable than that!

Even an elite school where 40% get As, is going to have a considerable variance from year-to-year even in large subjects like maths. In smaller subjects with 10 students per year, the model proposed to award grades according to the past 3 years' results. In the past my son's school got for music 100% As some years, and in other a mixture of A, A, B, C. Unlucky for you if you were in the all A cohort.

It's astonishing how anyone with even a GCSE in Maths could think any of this could make sense. It's the most basic principle of variance.

Surely they couldn't really be so incredibly stupid?! Was it all a trick so that those given unfair grades by their teachers would not complain about that, because well at least you weren't awarded a D by ERNIE.

But I think that just PROVES that our educational system is all messed up and how you perform on a few days during May/June one year should not affect the whole of your future

I absolutely agree

Ofqual tried to get the academic statisticians they needed to help them to sign very restrictive 5 year NDAs and naturally they refused. Why this was deemed do necessary is a mystery.

The problem with their algorithm is it didn't take care of the outliers properly and also created cliff edges for students at critical places in their class rankings where grades could rapidly be downgraded as one student could represent a higher % of a class than was allocated to one or more grades. This is why we saw some students massively downgraded.

The failures to recognise and correct these issues was either hubris or incompetence. Surely someone in government must have realised that these impacted outliers would make a very emotive and powerful counternarrative to the government claims that the algorithm was fair. Instead we have ended up with an embarrassing u-turn and very inflated grades based on predictions and a potential uni admissions and finding crisis.

Well, at least we they didn't scrap a national public health body in the middle of a pandemic and ask an ex-mobile phone exec wife of an MP who has no public health expertise whatsoever and has fail to deliver the test and trace capability properly to lead the organisation that has replaced it rather than an experienced public health expert.

@LizzieMacQueen

People need to stop putting asterisks in their posts, unless they mean to highlight them. So that A grade that is higher than an A, can you call it A+. Please. Then it doesn't stuff up the format of your OP.

I started doing this yesterday as was driving me nuts!

The algorithm was floored Because anything that's based on completely different data sets and trying to create an equal outcome for each isn't going to work.

@chomalungma

I wonder if this will be on More or Less tomorrow?

I love how small numbers can be deliberately misinterpreted.
I remember a dramatic headline in the local paper about bike thefts being up by 1/3 in a year.

There were 3 in 1 year. Then there were 4.

Didn't a mSM paper start something like Gaby Covid.

A list of towns who had doh led cases in a week.

Most has gone from 1-2, 2-4 and the worst one 4-8!

They had the second wave eating us alive as a nation

Why couldn't the results be compared to the general results for the school. By November every year we have the results as a % A to C, A to E etc. for individual subjects and as a whole cohort. I know my schools results were submitted by teachers, basing grading on mock sat in January, plus teacher knowledge of student - ie has worked socks off since January to improve etc. When they were all analysed by our Data Deputy Head our percentages were similar to the last few years with no grade inflation.

My school had 43% of grades reduced by 1 grade and a further 7% reduced by 2 grades. This means overall we have performed significantly worse than the last 3 years - how is that fair?

Given the extra month they had surely they could have used the total grades run the data they do for November and then delve deeper into the schools with massive inflation.

Don't think any schools had massive inflation. The subjects within schools which didn't have enough data to allow the algorithm to use went either with the teacher's grades (which does result in inflation because it's reasonable to predict 10 As because you have 10 A students, even though, say only 6 students will actually get the A* because someone had a bad day or whatever - you just don't know which ones), or with a blended approach in cases when there was a slightly greater amount of data.

Generally every subject in every school where there was a non-small cohort would have been given results that matched their 3-year average. Even though anyone could tell you that half the population is below average and half the population is above average.

Ofqual tried to get the academic statisticians they needed to help them to sign very restrictive 5 year NDAs and naturally they refused.

Confidentiality/Non disclosure agreements are standard for most jobs. Most will be forever, not just for 5 years. If you worked for a bank, you really wouldn't want the bank staff disclosing someone's savings at all, whether 1, 5 or 50 years. Same with whoever has custody of the KFC coatings recipe - that's also confidential forever.

Why would anyone think it's OK to go around disclosing confidential information that they've gained from their job? Why would anyone refuse to sign such an agreement? You go and do your job, you get paid, eventually the job ends and you leave. I must be missing something!

I can see why they wanted them to sign NDAs - without them, they knew that they either had to do exactly what they said or face the headline 'Ofqual IGNORED my recommendations, says top statistician'. Again, I think the problem was that Ofqual weren't set the job of coming up with the best algorithm, they were set the job of coming up with the algorithm that best 'maintained standards' (ensured that grades were as close as possible to previous years) and that was an awkward but unchangeable fact that they didn't want discussed and (quite rightly) criticised publicly.

Cohorts had to be 4 or fewer students to get the centre assessed grades.

@user1497207191

Yep you are missing two important points:

1. your examples are all in the private sector and not in the public sector. Such gagging clauses in the public sector are there to directly obstruct transparency and allow organisations to dodge FoI requests - not what you want from government bodies.

2. More importantly, the offer was not of paid work but to sit on an independent technical advisory panel. You can't expect an independent set of experts to be on an independent panel if you make them sign gagging clauses that stop them from demonstrating that they are independent, especially when you aren't paying them!

You are of course right. Ofqual didn't want these academics pointing out that they didn't take their advice or any flaws in what they were proposing. However, instead they just got criticised for saying that they didn't even listen in the first place, and their flaws were exposed anyway. There wasn't anywhere for Ofqual to hide here, they aren't a bank or fast food outlet. They should have taken the advice and done a better job as a result.

Teacher assessed grades are incredibly inaccurate and historically massively over-optimistic.

Coming up with a system to award perfectly judged grades based on flawed data is basically impossible and is going to always be unfair to someone as there is no way to manufacture perfect results from imperfect data. The only way you can do it is make certain assumptions to improve the accuracy at population level, but this will still mean certain individuals are given the wrong grades.

With the current U-turn, the problem has been handed over to the Universities who now have far more students meeting their offers than places available.

Where is your evidence for your first sentence star?

www.ucl.ac.uk/ioe/news/2016/dec/call-university-applications-overhaul-report-reveals-just-16-predicted-level-results-are-correct

econpapers.repec.org/paper/uclcepeow/20-14.htm

OK, UCAS predictions ( I wish they would change that word) are not predictions.

It is the same evidence produced and parroted out. there is no other evidence that teachers over predict and UCAS is a very particular set of circumstances.

Teachers are employed by exam boards every years as markers and moderators.

I also note that the common repeated theme is that we overpredict as if we are a bunch of softies. your second link shows that under predictions also exists.

No one can foretell the future accurately 100% of the time.

The CAGs went through various processes, as set out by the DfE : one of their requirements was that we give a balanced and full assessment of students and look for the best possible performance, with instructions to guard against under prediction of certain groups.

Commons select committee (in June) complained of the unfairness of the Ofqual system, Govt ignored the report.

But its now Ofquals fault? Really? Govt set the parameters for their algorithm.
Govt is proving to be lazy across a range of issues.

But the exams should have been sat, surely not too difficult to have come up with a suitable SD plan.

Using teacher predicted grades has led to massive grade inflation this year, so it's obvious that they do on average over-predict. Not all teachers do, and some even under-estimate, but as a population they do.

It's perfectly understandable that they'd err on the optimistic side for their students and any pupil is likely to be far better served by a teacher with a more positive view of their abilities.

The main issue with sitting exams was foreseen to be potential high levels of absence both beforehand and during. It would have been a bloody nightmare.

Essentially we are a country now overly wedded to terminal examinations as the gold standard. Gove's reforms created their own problem.

Hasn’t it been said eleventy billion times That CAGs and not the same as UCAS predictions?

It has.

They are also not the same as individual teacher predictions...

I think it's rewriting history a bit to say that exams could and should have gone ahead. That seems like a sensible thing to say in a world when you know how restrictions had eased by June/July; in March, when we were being told to only exercise once a day and only go to shops for essentials, it would have been (quite rightly) met with incredulity. It also would have been massively unfair on students because it would have been obvious that they might still be cancelled and so they would have been left with total uncertainty. Finally, it would have been hugely inequitable because some schools would have found it easy to keep teaching online right up to the exams and others would have struggled (or struggled to actually make it accessible to their students because of equipment etc), and that would have largely been a private/state school divide. Knowing now that not holding them caused such a mess makes it look less so in hindsight, but cancelling exams and doing so early and decisively really was the right choice at the time.

@Piggywaspushed

It has.

They are also not the same as individual teacher predictions...

Cool

Thought maybe id missed something and that this had NOT been mentioned many many times

Gosh the relief

Art is a subject which has long had teacher assessment at the heart of its assessment.

As I understand it, the work can be sampled, to be seen by an external examiner. If a discrepancy is seen, there is a process via which grades for that school are adjusted.

What the process needed this year was a large scale and targeted version of this.

CAGs could have been run against a statistical model of the last 3 year's results.

Schools where the result of this looked unusual (either positively or negatively) could have been sampled, with evidence of work for some students being scrutinised (there were, after all, a lot of people who usually work as examiners NOT being employed, who could have instead been used for this sampling work, trained by being given agreed standardised portfolios of work). If the scrutiny of sampled work revealed that the school's grades were significantly out of line, then the cohort in that school would have their grades adjusted.

Those examiners trained in the scrutinising process could also have sensibly been used for the appeals process.