to wonder how exam boards/ofqual can be so shit at statistics and if they even passed A Level maths themselves

[ShootsFruitsAndLeaves]

Roll a dice 30 times. How many sixes did you get?

www.random.org/dice/?num=30

I tried ten times:

5, 4, 6, 5, 6, 2, 7, 7, 4, 4.

Now imagine that's a comprehensive sixth form sitting an A Level. Even if we know the school's results for years and years, and we know 1 in 6 get As, then in any given year there are likely (99%) to be between 2 and 8 As in that class.

If you want to award the same number of As as last year, you're going to need to hand out 5 As to that school. But this year just by randomness maybe only 2 deserved it. Lucky kids. Or maybe 8 deserved it. Ouch.

Now imagine if that was a smaller class only 15, and normally on 1 in 15 get As. If student performance is randomly distributed then there's a 99% chance of 0, 1, 2 or 3 As. So you can have 3 A*s in a class that normally gets one. Not every school would be like that. But some classes in some schools WILL be in that position. And it's in fact NOT randomly distributed. Some schools will have good intake years. Maybe a genius physicist's twin children joined the school and are sitting A Level Physics. So the exam results are actually less likely to follow the past distribution than even random dice would be. It's even less predictable than that!

Even an elite school where 40% get As, is going to have a considerable variance from year-to-year even in large subjects like maths. In smaller subjects with 10 students per year, the model proposed to award grades according to the past 3 years' results. In the past my son's school got for music 100% As some years, and in other a mixture of A, A, B, C. Unlucky for you if you were in the all A cohort.

It's astonishing how anyone with even a GCSE in Maths could think any of this could make sense. It's the most basic principle of variance.

Surely they couldn't really be so incredibly stupid?! Was it all a trick so that those given unfair grades by their teachers would not complain about that, because well at least you weren't awarded a D by ERNIE.

"I think it's rewriting history a bit to say that exams could and should have gone ahead"

In Germany, the roughly equivalent exams went ahead in April, the usual month,
even though this was just after peak deaths

Nothing was open then except essential grocery and chemist shops

However, the exams were thought important enough

• and Merkel as a scientist would know that models & algoritms can't predict individual results -

so exam halls and gyms were deep-cleaned, desks put well apart, well- ventilated

Students attendance was very high, because it was their future at stake
Shielded students had extra precautions
Invigilators were volunteers, so no shielded ones

I'm sorry but I think that's total madness (and Germany, not the UK, was very much the outlier in this). It's not just the sitting of the exams, but the travel there, the psychological effect on students, the completely unequal access to teaching that they would have had.

Perhaps I think prioritising exams to this extent is so crazy because I actually think exams are very flawed as a pedagogical tool and as an assessment of ability anyway - I think it's amazing how many people are saying 'well you have to expect inflation because in a normal year a certain percentage of students would do badly due to having a cold, sleeping badly the night before, panicking on the day, etc' and how few of them then say 'obviously it is fucking bonkers that we usually use a system where any of those things might directly impact on the course of your life'.

It's astonishing how anyone with even a GCSE in Maths could think any of this could make sense

As a maths teacher I can tell.you that it really isn't astonishing however we don't want people with just a GCSE in maths making these calls. We want the nation's best statisticians, surely?

So YANBU and I can only think that there is some purpose to this. Overall it's almost certain that history will show this year that disadvantaged students were even less likely to get into elite universities than they normally are. I'm no conspiracy theorist but I'm actually not convinced that's an accident I'm afraid.

@starfro

Using teacher predicted grades has led to massive grade inflation this year, so it's obvious that they do on average over-predict. Not all teachers do, and some even under-estimate, but as a population they do.

It's perfectly understandable that they'd err on the optimistic side for their students and any pupil is likely to be far better served by a teacher with a more positive view of their abilities.

Not at all.

Teachers can assess a child and say where they are and give a very accurate representation of what a child can achieve.

What they can't do is predict who will be ill, who will mess up a question etc.

People really need to distinguish between the processes and use the correct words.

Not a statistician. But the model wasn’t there to generate variance as the harsh truth is that there isn’t much variance in the system. But the ranking of pupils, the allowance for schools with small classes in subjects to use the CAG grade (it was something like 1 in 20 private school pupils were in this size of class as opposed to 1 in 60 state) and the fact bright kids could be impacted disproportionately by using the school’s previous record in state schools all skewed the system. But overall there was a similar set of results to 2017-19 and the Uni system did it’s usual job of adjusting offers and putting things into clearing, as well as lots of kids still holding UC offers nowadays. As it happens I work in one of the selective subjects in HE - we filled off Ofqual grades given - we don’t always. Now we will have a surplus of students who met their offer.

The flaw in all this is that the decision was made by statisticians.

There is nothing about this year that is comparable to last year, three years ago or even next year (which if it isn't addressed now will be a bigger clusterfuck).

Many people will have looked at the algorithm and saw it disproportionately impacted poor bright students and those at the lower end of the ranking. But they all thought that was ok.

Overall it's almost certain that history will show this year that disadvantaged students were even less likely to get into elite universities than they normally are. I'm no conspiracy theorist but I'm actually not convinced that's an accident I'm afraid.

Tbh , you don't need history to tell you this ; it's already the case. the stat the government is peddling about 7% more masks a great deal.

Interesting article in The Guardian yesterday about our whole notion of entrance requirements and elite universities, on the flip side.

It also affected people at the top in most schools. Before CAGs were restored, nearly all the As in my schools (large , fairly high attaining comprehensive) got downgraded to As and some to Bs. Only the subjects with nationally higher likelihood of getting an A kept them.

I'm no conspiracy theorist but I cannot get away from thinking there is something more at play purely based on the fact when Gove made education a cluster fuck his advisor was Cummings. When it becomes a cluster fuck again, GAV is out front, Boris is hiding in a bunker on holiday and Cummings is still working and probably running the show.

And we all know he loves data and a statistic.

Cohorts had to be 4 or fewer students to get the centre assessed grades.

Not exactly. It's quite complicated, but there were three options:

CAG only
Model only
Blended CAG & model.

With little (2 or fewer in last 3 years) or no history CAG, is always used, regardless of cohort size this year. And the cut-off point for pure algorithm was around 10 students per year.

So a school which teaches for example Music Technology and has 6 students per year will use a partial blend of CAG & algorithm. So llower than CAG but higher than algorithm.

BigChocFrenzy
In Germany, the roughly equivalent exams went ahead in April, the usual month, even though this was just after peak deaths.

Several articles from 2019 indicate that only 30% of German school leavers sit the Arbitur exams - much lower than the percentage of UK students taking A-levels;

"The Abitur is the qualification that pupils get after the final two years at a Gymnasium, a selective state school comparable to a UK grammar school, which is designed for supposedly more academic students. Around 30% of German secondary school pupils attend Gymnasiums."

and Merkel as a scientist would know that models & algorithms can't predict individual results

You can't please everyone;

"German secondary school state exams continue despite opposition."

"Students across Germany have written open letters protesting against the decision: as well as health concerns, they cite pandemic psychological pressures–from struggles to study to fears over parents who have lost their jobs."

"Previous grades"
"Continuous assessment and project work mean the written Abitur exams count for about 30 per cent of any student’s final mark. According to analyses of previous year’s results, grades fluctuated minimally when the written exam results were added to the previous grade average."

"Artemisa Ruiz Bustos, whose daughter is sitting her examinations in the Nelson Mandela School, says health concerns linger over the examinations. “We had two children in the exam class with suspected Covid-19 cases,” she said, “but because they were not in a risk group, they weren’t tested.”

"Germany’s teaching union GEW said it was unhappy with the decision but had no means to instruct its members not to supervise the exams."

"It argued the staggered nature of the Abitur examinations–some states are already finished, some have yet to begin–left some students at a disadvantage with a longer period without regular schooling."

“The ministers wanted these exams by hook or by crook, and have created an unnecessary psychological burden for students,” said Ulf Rödde, a GEW spokesman. “The decision betrays their narrow ideological view of education, not as a learning process for children, but one focused on exams.”

"That echoes the views of many at Berlin’s Nelson Mandela School, said Paola Aliverti, of the school parents’ committee: “I think it has to do more with the German love for the piece of paper than any sense of responsibility for students and their future.”

www.irishtimes.com/news/world/europe/german-secondary-school-state-exams-continue-despite-opposition-1.4250659

With little (2 or fewer in last 3 years) or no history CAG, is always used, regardless of cohort size this year.

Not sure what you mean here?

If you mean there was only a history of two years of prior results , so CAG was sued, I am afraid you are wrong there OP.

My subject had 8 students and only two years of prior results and the CAGs were not used. There may well have been slightly less algorithm use as a result (ie begrudgingly allowing one A*/A even though this looks like 12.5%!), but it was not all CAG.

Eton had no historical data for one subject (so their head said) and so the national data was applied.

The whole problem with the algorithm is it didn't allow for the realities of changing results.

It made lots of assumptions based on grades but the reasons were ignored and often it's the reasons that effect results.

For example the 4 secondaries schools near me. Pass rate for 4+ maths and English 2017-19

School 1 63% 68% 73%
School 2 51% 50% 55%
School 3 73% 68% 63%
School 4 78% 79% 80%

Each school has a very definite cohort type year on year and specific ethos which affects population type and results.

If you mean there was only a history of two years of prior results , so CAG was sued, I am afraid you are wrong there OP.

No, I mean two total students in past 3 years.

Anyway new process is also unfair.

Ds' CAGs almost identical to algorithm grades. Other schools will have inflated massively. Those who inflated will be rewarded for this.

@holdingpattern

I seriously don't get how no one understands the algorithm or purpose.

If School A for the last 100 years has only managed
AABBBCCCC
AAABBBCCC
ABBBCCCDD
BBBBCCCDU

Then this year the teachers say

AAAAAAAAB

I think everyone would agree its quite amazing how the pupils of 2020 were so brilliant.

The algorithm was there to say - normally even at your best, you never got those results. We will be generous and add inflation in, and give you
AAAABBBBBC

For better schools, they were stricter and hence their downgrading looked smaller. For less better schools, they overestimated greatly.

For individuals there was always the possibility of a star pupil. But if teachers had ranked them 1, then they would have still got the A grades. If teachers ranked them 8, then they might have found themselves dropped down.

What has happened is teachers had to rank pupils too.
So in many cases of people saying but I got A in prelims, and Joe got a B in mocks/prelims but Joe got awarded an A and I got a C. That's because the teachers (unbiased of course) ranked each pupil and you were ranked below Joe, both considered an A by the school. Ofqual had to readjust these predictions and you dropped to a C because of ranking.

The other thing everyone seems oblivious to, is that every year the exam results are normalised.

If everyone got 80% in maths or higher, they move the pass mark to 90% on the basis the exam was too easy and they have to normalise the results. So in the past 100 years if there was a year that everyone got 80% and if you could see marks vs grades, you would find that at 80% would have got a D.

This year by awarding 40% inflation, Universities will not have enough places and will have to select on the basis of something else. Maybe hair colour, maybe an essay, maybe like LNAT and UCAT tests for law and medicine. But they will have to select somehow.

The unfairness was cancelling the exams. These were the closest to giving a fair comparison. Teachers are different, schools are different, mocks, tests, prelims, marking and setting are different.

Except, especially at GCSE, but to some extent at A level, for optional subjects, the cohort varies hugely each year, in schools with very mixed, non selective intakes. So to peg the results to previous attainment can be wrong.

I would say OP that I haven't as yet found a school with massively over inflated results across the entire year group. I don't think the maths works like that.

What do you mean the maths don't work like that?

If school A had 16% A* last year and handed out 18% this year, while school B had 16% and handed out 30% this year, then school B has massively inflated grades. That's all there is to it.

@ShootsFruitsAndLeaves

What do you mean the maths don't work like that?

If school A had 16% A* last year and handed out 18% this year, while school B had 16% and handed out 30% this year, then school B has massively inflated grades. That's all there is to it.

Or they had different pupils in that subject.

@ShootsFruitsAndLeaves

What do you mean the maths don't work like that?

If school A had 16% A* last year and handed out 18% this year, while school B had 16% and handed out 30% this year, then school B has massively inflated grades. That's all there is to it.

Surely it will depend on the nature and previous attainment of the cohort. Say the subject is French, and when this year group was in year 10, a new French teacher arrived. Initially, they taught GCSE, and really inspired the cohort, with the result that those kids who got As / A*s at GCSE French decided that yes, they would do a language A-level, whereas before this teacher came, they would tend to do another Science subject, or politics to go along with History, or maybe Spanish.

So this year's cohort had 40% of pupils who had As / As at GCSE, whereas normally it would be 20%. It would be entirely reasonable that the percentage of As in the final A level grade is much increased, without having to suggest 'massive grade inflation'.

Equally, in a school where there is significant arrivals and departures for A-levels, it may well be that the balance of who stays and who leaves changes quite dramatically between years.

Or just cohort to cohort variation - DD's cohort was very much less able than the school's norm, which both depressed the GCSE results AND meant that many more of the very ablest students left to go elsewhere. DS's cohort, on the other hand, was the most able in his school for many years, and almost all of the most able students stayed put for a-levels because their experience was of being in a group where learning was valued and there was a big group of highly able kids..

I was taking issue with the word massively and thinking on a per pupil basis ( your justifiable issue with the algorithm seems to be partly that this case by case approach wasn't done?).

Awarding lots of students a B rather than a C because they could get either, would look like massive grade inflation at B on raw stats but for each child it isn't. (if DfE can claim that students losing out by one grade isn't a big deal, I can say going up by one grade isn't either!)

One or two subjects within a school might have over awarded at a certain grades and that also could affect the whole schools' headlines but does not mean the school somehow recklessly hyped their results.

I am a bit disappointed after your opening posts that you have now gone to blaming teachers for some kind of wild (and concertedly deliberate) optimism.

"If school A had 16% A* last year and handed out 18% this year, while school B had 16% and handed out 30% this year, then school B has massively inflated grades. That's all there is to it."

School A has 50 students studying this subject, 8 achieved a high grade last year, 1 extra got the grade - 2% increase.

School B has 7 students, 1 achieved high last year 16%, this year 2 achieved high, 30%

Unfortunately there are just too many variables to assume that %increase is over inflation of grades or should be used as the only factor.

I would have liked to see all anomalies should have been pulled out and reviewed and justification/confirmation of agreed grade.

Unfortunately there are just too many variables to assume that %increase is over inflation of grades or should be used as the only factor.

I'm using inflation in the mathematical sense. Whether or not some of these were justified is possible. But on the whole those whose schools inflated grades have an unfair advantage.

disadvantaged students were even less likely to get into elite universities than they normally are

Work in a Russell group that went out of its way in clearing this year to adjust for WP students. So I’d need to see the final evidence for this.

Totally agree can't.

I posted above showing 4 local schools and how some have remained constant and 2 look to have done a switch!
But actually in that case I can confirm that school 1 despite getting better 4+ pass rate year on year has remained below average in progress 8 whereas school 3 who's results dropped actually had better progress 8 each year.

All based on cohorts and entry level of the students.

Plus school 3 despite getting less passes actually has a much higher percentage of 7-9.

Cohort isn't as simple as fitting into an algorithm. It's so much more sensitive than that.