ISEB exam results - bad day or system botch up ?

[ElonanotAlone]

Do you think the ISEB results truely reflect your DC's actual ability? Just that really.

We have been fortunate to receive exam feedback from a couple of Schools with quartile cohort results and eventually our actual scores from a sympatheic and understanding school. As a consequence we have been shocked by the inconsistency of ISEB results versus our DC's known competency on a particular subject, notably English and Maths.

Our view is based on known performance on a number of verifiable asessments - a non ISEB exam eg Grammar, mocks, school exams, CAT4 outcomes, etc or teacher expectations of performance.

We have tried endlessly to rationalise the outcomes, perhaps DC had a bad day, greater competition, however, in discussion with a few at school others facing similar outcomes, we have come to ponder, was there a glitch or ISEB malfunction with the exam format questions/answers or set up issues within the inviligation centres?

Hence, curious to see if thus is an isolated school issue or one felt amongst other parents feeling disillusioned with 11+ choices we are forced to accept..

@LondonMummer I don't really know but having gone through 11+ with my other kids I do think it's always worth going the extra mile on vocab and English as it's about being compared to others and, generally speaking, boys at that age are probably generally weaker in comprehension whilst there are lots of boys excelling in maths/NVR.

@Teacakes100 Ha ha might seem that way but, no, I neither work for Atom/ISEB or receive commission. Tbh, my son would probably have done well anyway, as we had a tutor and did paper tests, but we did use Atom because of the adaptive feature. Mainly custom and mock tests. Looking through the results he would often have lots of incorrect answers but still get high standardised scores so he must have unlocked those harder questions early on. He probably used it on and off (i.e. not hitting the weekly target) for 10 months and went from 115-135-140 but then some of that would have been the other stuff we were doing and maturity. We also tried a Pretest Plus type online mock. That was much harder - an eye opener.

I'm going to ask one of the schools now for the full results, so we shall see.

I do think this year was unusual in more ways than one. Due to so many schools using the ISEB, many many more of those who would normally only sit state 11+ grammar tests - and perhaps 1 indie at the most - signed up to multiple (ISEB) schools as it would mean only taking the test once.

So, all of a sudden, those kids coming from private or prep were competing with a much large cohort of kids from the state sector, some of whom would have spent years prepping for the 11+.

I've had experience of both state and private and, surprising as it might seem to those who have only been in private or preps, there is a lot of very good state schools achieving superior results in English and Maths (though much more limited in terms of all-round education) with some super bright kids attending these.

In reality, many of these children will be the ones scoring in the top 2-5% nationally in the grammar and super-selectives and will likely choose these schools come offer day, and free up spaces elsewhere.

That's not to say that the ISEB would fail some children, just as some traditional indie 11+ written tests would possibly have favoured those coming from prep/private backgrounds (particularly if you look at creative writing/essays). Ultimately, those kids prepping for super selective state schools have likely been cramming for years and some would have sat multiple mocks and 11+ tests in Bexley, Essex and Kent before the ISEB. All good practice!

If you go back some years, there are data on the 11+ results from e.g. Bexley and Kent with details of which schools the kids attended. For some of our local preps, my friends were shocked to realise that their schools did not produce very good results compared with other preps or high achieving state schools.

@lightattheendofthetunnel2021

Oh so funny, I was laughing so much last night when I read your comments,....I have woken and am still laughing 😂

Your clearly either from ISEB , Atom or you endorse them...

Mums can tell......😉😁

Well done @ElonanotAlone and all mums on here....so the cover up is now in full swing, let’s spot the next one....😎😎

No one at our prep is talking - they are all still getting used to the outcomes of the chaotic process this year
My child did okay - I didn't ask for results but the interview feedback on his ISEB was as expected
Others did not fare well and people have been scrabbling around for places and doing late applications to alternative schools

I suspect more kids applied to private secondaries this year and many who would normally go state have decided to switch due to lockdown learning - the lack of waitlist movement may bear this out

We used Atom this year and found it to be very poor. After a few goes we abandoned it. Our DC was finishing tests very quickly, it wasn’t adjusting itself, and we didn’t trust it at all. I suspect it have many people a false impression. We went back to Bond books and some Bofa, both of which were much better.

We didn’t rate Atom and cancelled it after 1 month.... we found questions or question styles kept repeating in subsequent mocks so found the variety of questions poor after the first few mocks.

English paper was just something else! The grammar terminology was nothing like what’s required till year 6. Even my older year 10 child had not heard of some.

It’s best to use wide variety of material - GL CGP books, bond books etc.

@Teacakes100

Honestly, cross my heart, no, no affiliation with Atom or ISEB, nor connection in any shape or form. Just our experience, for what it's worth. DS did say that he thought the ISEB was much harder than (and not that similar) to Atom. Some of the English on Atom he found quite difficult. I left him to do Atom so that suited me, but means I didn't get a sense of whether questions repeated, may well have if we'd done more on it but he never finished all the subjects.

We had a tutor and did the GCP and Bond books too. Definitely found the onsite mock tests useful.

What I do find frustrating, however, is that we don't get access to the full ISEB data. From the 11+ grammar tests elsewhere, you could request detailed data e.g. raw scores, ranking etc etc (often a response within a couple of days). And we didn't ever pay for those!!

Given that we pay indirectly for our DC sitting ISEB, I think the least we should get is a summary of the scores. I do realise some schools will weigh certain subjects differently than others but as long as they explain that we should, in my view, be entitled to the data.

@DonGray

No one at our prep is talking - they are all still getting used to the outcomes of the chaotic process this year My child did okay - I didn't ask for results but the interview feedback on his ISEB was as expected Others did not fare well and people have been scrabbling around for places and doing late applications to alternative schools

I suspect more kids applied to private secondaries this year and many who would normally go state have decided to switch due to lockdown learning - the lack of waitlist movement may bear this out

Yes I definitely more kids applying for private senior schools who would normally only look at state secondaries (selective or comprehensive schools). Interesting point about the lockdown remote learning provision - I actually know a couple who moved their DD to private because they felt their usually excellent state schools just couldn't compete.

Maybe it gets tricky as decisions are not made solely on scores unlike grammar?

We didn’t apply to grammar schools as too far. The only indication we have is Atom mocks and getting into the school we preferred. But I’m ok with that really. But maybe I’d want to know more in a different situation.

True and although we got the school we wanted too, I still think the data should be accessible and available. Maybe this will ultimately lead to independent schools having to be more transparent? Some are, some aren't...

A happy Mothers Day to us all !

I actually design and manage complex algro's for large financial organisations. So hopefully my insight might be of some modest value. My DS got an offer as you all know from the school we wanted, but also as you know feed back was that one section was weak, a section which DS is without doubt very good at in over the last 2/3 years of prep-school testing. So that has make me question the data, as so many of you have, not only as a mum but also professionally.

Interesting that you discovered @Drminime that Atom manually rebalanced data from Vietnam, because we noticed issues with Atom. So as a team we set up a series of dummy accounts with Atom and worked out pretty well the algo behind it. But always found further issues, the conclusion we came up with was that there must me a manual process somewhere as well. So well done @Drminime.

On to ISEB and the issues some have experienced. My take on this is that the error has occurred in one or a combination of areas. But this will not have had an impacted on all childrens results. (I have written this in the most simple of terms, so that terminology does not confuse)

(1) Some of the questions in the question bank are not age specific, it is 13+ instead of 11+ or has been incorrectly sub divided in the question bank pool of data.
(2)This leads to the adaptive randomiser pulling up questions from the pool that is incorrect for the test.
(3) Incorrect recording of some pupils selected answers to questions, this is likely, as there was a vast increase in users this year and how the interface would work will be prone to issues.
(4) When they ran the data analysis the algo corrupted some of the data, giving false outcomes, again likely and happens.

In a algo, that manages financial products, we are obsessed with focusing on the error factors when we are back testing. As even the smallest error has an enormous financial impact.

For other organisation (non-financial) they will never have the resources to invest that much money, man power or operating system capacity to ensure that there are no errors....hence @Drminime stating that Atom were using a team in Vietnam..it's cheaper!

So what is my conclusion?
There is always a correlation within possible outcomes. In this case the CAT4 scores that prep schools have or the internal data that schools have with teachers professional appraisals should be very closely correlated with the ISEB data. Even allowing for a 'bad day', that difference should not be more than 5% or 5-6 points and over an average of the the 4 ISEB sections less than 5 points.

What some are experiencing is vastly more than that, that does not happen in a well managed process which has been back tested. Some one here said that the ISEB team were working remotely. That combined with the huge increase in demand in a very short period of time, has clearly led to a processing issue for some pupils.

ISEB do not allow any transparency nor are they accountable to any regulatory organisation. Yet it seems for some, many senior schools have take ISEB's data as gospel and ignored all the tried and tested available data supplied by prep schools.

@Daydreamer85 @Sterou
thank you for a comprehensive post that details where and how the ISEB algorithm may have gone wrong.

I had not even considered your point 3 and 4 above, but clearly extremely valid and of concern.

@ElonanotAlone @DecisionToMake123 @MulleWine20 @walking2020 @Mimijoyx @RamonaRevival @Sterou @tryalittle

How do you get the iseb results please.
Thank you!

One of the schools we applied just gave them to us

When can Head plead to senior school inconsistent iseb results? After parents got “no interview” email? Or before the email?

Can I ask, what were the scores the children received for the ISEB and for their mocks? When they did mocks, was it adaptive?

I am also REALLY nervous about this. My dd thinks she bombed the first adaptive exam she did. Her mocks range from 116-142 (maths are the ones in the 130's-142, the other ones are 116-136 range).

@SamPoodle123 those results sound quite strong. It is not the same as CAT4 scores, because the scores are standardised based on all ISEB candidates, which is not the same as the general population. So a CAT score of 130 means your child is in the top 2% of all children in their ag group whereas an ISEB score of 130 means your child is in the top 2% of all children in their age group who took the ISEB that year. That is why the score is very useful for the senior schools - they can get a percentile rank of the candidate compared to all the other applicants.

So it is more like the Atom mock Scores, you take the score and add about 8 points to get a comparable CAT score. I'd say any score 120+ puts you in a very good position for selective schools. All your daughter's scores sound very strong, 116 might be a problem if it is in English as some highly selective schools will want an ISEB score of at least 120+ in Maths and English.

@swgeek many thanks for your feedback. My dd never took a CAT test. I wish they did them at her school. So we have to rely on her Atom scores. I am really hoping she managed to score well enough. Her English and Verbal are weaker than her Math and non Verbal, but she has been in a French bilingual school since age 5, so literally half of her schoolwork is taught entirely in French, meaning she loses out a lot on English Vocabulary as half her school life is taught in French. Lessons are taught one week in English and then one week in French. It is pretty amazing though, as she is entirely fluent in French, whereas we only speak English at home. However, not very helpful for the 11+ English and verbal! I am really hoping the schools might take that into account, but doubt they will! I am hoping she had a good day when doing the exam and got a 120+!

Hi SW - when you say add 8 marks to atom to get to CAT how does the correlate with ISEB? My DS scored 128, 121, 110 and 108 in his ISEB, average 117, and we're wondering if he's flunked it.

@HandyGirl76 I was told aim for scores above 120 for ISEB with super selective schools. We aimed for this and our dd got interviews at all 4 schools she applied to including G&L. Now, we are waiting to see if she gets offers :) Crossing fingers.

We did the test during covid year. At a state primary, so no cat scores or tests.

On atom, my son was never getting less than about 122. Usually hovering around the 127 mark all round. Sometimes (maybe half the time) he’d score in 130s. Sometimes 140s. Sometimes low 20s. I think the pp is right: Iseb, on atom at least, aim for 120 plus for the SSs (grammars and indies)

you see, with his scores he actually got various academic scholarships, including to one of the very top 3 indies, and a place at a SS grammar. I wasn’t even sure he’d pass! I know that sounds nuts now, but I really wasn’t sure; plus I wasn’t sure how accurate the atom tests were. But atom, at that point, said to add 6 marks to get the cat score.

Iseb scores in the high teens, go for schools like Dulwich or whitgift or Putney high for girls. Might swing a place at less difficult grammars. But to a pp this is not flunking, this is still doing really well!

scores around 110, maybe emmanuelle, St John’s, etc.

worth remembering that a child will feel happiest with children who learn at the same pace. So if your child fails whatever school, it’s actually in their interest. The pace at my sons school is SO FAST!!

@Itci that is very good advice. My dd is also from state school, so we had no idea her ability or what school for her. We applied to a range G&L, Putney, Emanuel and Francis holland. She got to interview stage for all and now we are waiting to hear back.

I was so nervous for the exams, as I was not sure the Atom scores were correct in reflecting her real scores...but it seems like they were. She even was certain she failed the Putney exam (for her that was the first exam and she said most difficult). But she passed.

I was not so sure bc sometimes she would get the 115 etc for atom in verbal/non verbal. English always above 120 and could go up to 142. Math was usuall 130-142s (several in the 137-142 range). I was worried about the verbal and non verbal dragging her down. But I guess it was okay. We are still waiting thought to see if she gets offers. Her interviews went well, so crossing fingers!

@SamPoodle123 It's so worrying!! But with those scores (as I now know!) she really ought to get places at EVERYTHING. Indeed, the interview invites demonstrate that she's got the right scores. Fingers crossed with the interviews indeed! (Oh and don't worry about them either. You just can't second-guess what the schools are looking for, which probably differs acc to school!)

@Itci thanks! I hope you are right! I am really so curious what they are looking for in interviews and if perhaps they pretty much know if you are already getting an offer based on the exams. How much weight is on the interviews?! I know my dd interviews well at least (she is confident, and I was told by our head she will have no problem w the interviews). Her last interview, the teacher came out shook my hand and said "She aced it!" He was very enthusiastic with her use of phrases and descriptions/use of vocabulary. I am hoping that means she got an offer! lol.