Hattie's effect sizes - class size doesn't matter and other bobbins

[noblegiraffe]

I keep seeing on here people referencing Hattie's Visible Learning and saying things like 'homework doesn't make a difference' or 'class size doesn't matter' when clearly the effect of those things can't be reduced to a single number.

Obviously whether homework makes a difference to achievement or not depends on the homework. A kid who diligently goes home and practises their times tables will end up with better recall than if they hadn't. On the other hand, a kid who goes home and makes a junk model of a volcano could very well learn little new about volcanos.

Clearly class size makes a difference. With a class of 15 I can get to know them better, provide them with better feedback, get around the class to talk to them more often than I can a class of 35. A class of 3 learning a language will be much more difficult for conversation than a class of 10.

Anyway, I just came across this article which details why Hattie's statistical analysis is crap and should be seen as pseudoscience. Can we bin Hattie along with VAK and Growth Mindset?
mje.mcgill.ca/article/view/9475/7229

noblegiraffe I have to confess that my stats date back to the last ice age so struggled to follow the detail but understood the summary.

I am interested in studies in education. The one I often quote is this:

educationendowmentfoundation.org.uk/resources/teaching-learning-toolkit

The class size one is interesting as it does seem that 15 is a sort of cut off where there is an impact. It would be interesting to see if it was subject dependent.

There has to be study, analysis of those studies etc. Educational practice should not be developed purely on the basis of anecdotal evidence. However, equally educational practice should not be slave to the latest theory to the exclusion of all others.

Class size isn't just subject dependent, it also depends on the type of class and how independently they are able to learn. It's why when schools set for maths they often have large top sets (you can set them off on an exercise with minimal intervention) and small bottom sets (generally need far more support). I really don't understand how primary school teachers cope with young children in the sort of class sizes we reserve for our most capable learners.

I agree that we do need research into whether interventions are effective or not, and that single studies on their own are not especially useful as they are very context dependent. However, Hattie doesn't appear to be very good at stats and the article suggests he should have consulted a statistician before writing his book. This also raises the question of how good the stats were in the studies that he was amalgamating. If someone can become that prominent in education based on dodgy stats, then clearly these sorts of things aren't checked very carefully in education research. There are millions of poorly run studies around the world for medical data - people run studies, write reports showing all sorts of nonsense like homeopathy works or vaccinations cause autism. There's no reason to think that the studies that he bases his research on are all high quality and meaningful even before he applies his calculations to them.

The education endowment toolkit appears more nuanced and perhaps more useful, but I am concerned that when I looked at their evidence for setting by ability (negative impact) one of the first references on their list was Jo Boaler, who is at the very least a controversial figure in maths education.

I've read a lot of criticism of various sorts about the EEF. It doesn't really seem to have entirely lived up to its promise to be rigorously evidence based. Here's one about its evaluation of Switched On Reading:

horatiospeaks.wordpress.com/2014/05/10/a-convergence-of-interests/

kesstrel I think the article supports noblegiraffe's point about the statistics.

Every time statistics are published it seems that the authors are looking for a simplistic headline: an initiative does/doesnt work. This, that or the other will solve a problem. All of this couched in suitably vague terms so that no part of the assertion is clearly defined.

Gnome Absolutely.

Learning Styles and Growth Mindsets are alive and well and living in schools near me! I blame the IOE, perhaps unfairly.

I haven't heard of Hattie but get your point. I'm happy to help DC with reading and tables but I loath the "build a scale model of a Viking Longboat from Twiglets" m'larkey.

I do love the word bobbins.

Even Hattie admits that 50% of the statistics in his book are wrong. And his work certainly doesn't support an argument that homework doesn't make a difference. It suggests the difference is small at primary level by quite large at secondary level. It is only by aggregating the two together that you can come up with ridiculous statements like this.

One of the things that annoys me is the way some leading educationists routinely reject research findings when they show that their favoured approaches don't work. Flawed "research" such as Hattie's does nothing to help the case for a proper approach.

prh47bridge that is the problem with getting too wedded to a theory.

What we always say to the DCs is look to see who paid for the research. Obviously research sponsored by the 'School Dinner provider's federation' is going to result in conclusions favourable to the provision of school dinners.

Here's Hattie's own website with his effect sizes in order, with 'homework' near the bottom. If his own work suggests that this sort of amalgamation is nonsense, then we definitely should be throwing him in the bin of bad ideas as he is not even representing his own work honestly.

Why do people do this sort of crap? And then we get humanities/PE-trained SLT lapping it up because they don't know any better.

Noble, what most people fail to take into consideration is effort from the student. We have CAT scores and KS 2 scores suggesting a grade More effort or less effort from the student and of course impact of teaching can change that completely. However students are still put in sets depending on this and practically there is little movement between the sets. Unfortunately in the case of maths it often means that able children may be at a disadvantage because of wrong setting. As for class sizes they make a rather significant difference , I agree. In a large class the quiet average children do not reach their potential as the teacher deals with the moveable or the less able in the group.

• More able not moveable

I really don't understand how primary school teachers cope with young children in the sort of class sizes we reserve for our most capable learners.

There are lots of things I don't understand about the differences in 'common practice' between primary and secondary, and think proper research into the differences would be really interesting.

For example, the vast majority of primaries don't set - many can't, because there's a single class per year group. Does progress of all ability groups suddenly accelerate once they start secondary and are placed in sets? Or does it only make a difference to certain groups?

Primary marking policies focus on much more regular, indeed commonly 'always for the next lesson of that subject', and also more extensive written marking. Secondaries - where the pupils are much more likely to be able to read the feedback - seem to have policies in which they mark much less regularly and, for most work before exam years, much more minimally.

Primaries place a huge focus on high quality, frequently changed 'classroom environments' - displays, working walls etc - while secondaries don't.

Parents fuss a lot in primaries even about jobshares, stressing the importance of continuity, knowledge of the child etc. Secondary pupils have a different teacher for each lesson of the day.

Having watched another cohort of carefully-nurtured, single-teachered, non-set, beautiful-classroom-dwelling, every-piece-of-work-carefully-marked-in-two-colours 11 year olds leave us for a secondary where none of these things apply, I wonder what magically changes over the 6 week break.....

Paperclip Mover is right to blame the IoE for the persistence of Learning Styles and Growth Mindsets. Their understanding of research and the stats supporting the conclusions derived is lamentably weak.

For instance, The IoE has endorsed Reading Recovery, and now Switch-on Reading, for years, despite the fact that measurable advantages are shown only in short term follow-ups. Looked at after an interval of a few years, the pupils given the intensive support of RR slide back to their previous relatively weak positions in percentile terms.

The McGill piece is brilliant.

If we could get proper RCT research in education, we'd be so much better informed. But proper research costs money.

What's odd about Hattie is that his publications seem to be confined to Visible Learning. Where are the triple refereed articles in academic journals: were the drafts rejected on the grounds of confused thinking and dodgy stats?

And the IOE is in London and trains teachers, some itself, and some via school based schemes. Then there is the in service training they provide.The majority of teachers in some London Boroughs have been trained by them in some form or other so whole schools are emeshed in RR, Growth Mindset, Learning Styles and no one to question it. Well, no one with authority who the school will listen to anyway...Ofsted don't care so no one cares...

So there is a vested interest in publishing research which supports a particular view. Means it isn't really research at all. Just some fancy tables and graphs.

Oh what a shame - the idea of Hattie's work was so enticing! Shame it is not as rigorous as it first appears. Although I'm not totally surprised about the publishing business - they print books that will sell - so they achieved their aim in this case!

In (slight) defence, my understanding was not that homework was completely useless. Instead I understood it was the practice of homework for the sake of it. Clearly work to enhance learning (times tables example above is excellent) is very valuable. But I think that came through in Hattie's commentary.... the headlines were misleading.

Someone posted above about the differences between Primary & Secondary. I've always wondered why professionals in different settings don't think more broadly so it is nice to see cantkeep being curious.

I'd like to see Primary school's be more aware of secondary practices and actively prepare pupils by slowly withdrawing some of the usual supports towards as the kids go through. It will create independence in most cases and highlight those pupils who may need need extra help in advance of the "lemming drop" transition into secondary that currently exists. I think there is a widely known dip in results post transfer so I don't know why this hasn't been addressed.

Our family experience with an SEN child is that Primary supports are added and added until the child is wholly dependent. No thought is given to removing those supports to establish independence and self reliance at a pace that suits the child in order to equip them for the future.

Id really welcome some proper evidence based practice in schools!

Cant

There are lots of things I don't understand about the differences in 'common practice' between primary and secondary, and think proper research into the differences would be really interesting.

I wonder if some of this perhaps comes from the still lingering influence of the Plowden Report? I know it was 50 years ago, and obviously lots has changed since then, but it made a big push toward making primary schools very different from secondary.

Part of that was that it seemed to want Year 6 to look a lot like Year 1, in terms of having pretty much the same kind of free-form, individualised discovery learning where children followed their own interests.

This is in contrast to some other countries there is a distinct boundary between infant school, or pre-school, and junior school, with an expectation that teaching methods will become increasingly more formal and traditional from age 7 or 8. For example, moving from sitting around group tables to forward-facing desks at that age.

This leads to a less sharp distinction between primary and secondary methods, IMO.

"For example, the vast majority of primaries don't set - many can't, because there's a single class per year group. Does progress of all ability groups suddenly accelerate once they start secondary and are placed in sets? Or does it only make a difference to certain groups?"

It's a reasonable point. I guess the reality is that the gaps get wider as kids get older, and the transition between primary and secondary is seen as being a convenient cut-off point for starting to put kids in sets. Based on what I have heard, though, higher performing education systems usually delay setting until 14 or so, not 11 (although they are able to, I suspect, in part because primary schools are doing a better job of teaching the weakest-performing kids, so that the gaps are less apparent).

As Kesstrel says, it's considered normal in the UK for primary schools to have a deliberately child-centered "look" in the way classrooms are set out and ideas about the teacher-child relationship. In countries like, say, France or Germany, children may start school a year or two later than British kids, but once they do start at 6 it all tends to look a lot more formal than we would expect for our own Y2 kids.

Hattie gives the impression of just not liking teachers very much. He comes across as scathing about teachers in general when he talks about education.

"Teachers should avoid becoming researchers in their classrooms and leave the job to academics, according to one of education’s most influential professors."

www.tes.com/news/school-news/breaking-news/leave-research-academics-john-hattie-tells-teachers

I agree that class size is likely to be less important than curriculum and teacher quality, but I would be surprised if it had no effect at all, especially in school systems where teachers have very little non-teaching time in their schedule, because it is going to be very hard to get all that marking done.

Another thing: looking at UK classrooms, what strikes me is that they often look physically overcrowded--too many desks crammed into a space which clearly was never designed to have so many desks there. Kids are all squashed up against each other, annoying and distracting one another and getting into each others' space, and it is physically hard for the teacher to move around the room easily, making it very hard to quickly stamp out crappy behavior by moving around to the offender and speaking directly to them. In school systems where bigger class sizes have long been the norm, there is usually more space between desks (and no group sitting under normal circumstances), and teachers often use microphone devices to ensure that their voices carry across the larger classrooms.

As for homework? As other posters have mentioned, the importance of properly-focused homework is probably being lost because Hattie is not distinguishing between homework that is, well, focused, and homework that is a waste of time. Costume-making and "bake a cake in the shape of a mosque" can go to hell, but there is no way that reading practice, times tables and extra maths problems are not going to improve performance.

Far easier to have a class of 15 than 35 as noble says.

If it wasn't an advantage, independent schools wouldn't bother mentioning small classes.

One form entry primaries may not set as such, but the children are generally grouped by ability for maths and reading- classic is triangles squares and pentagons; although the new curriculum and aporoach seems to be to allow children to pick their own level.

MaryTheCanary
Children in France do not start later. Children are legally required to be educated in France from the year they turn 6, but the vast, vast, VAST majority of children start maternelle the year they turn three. There is huge social pressure to have children in maternelle and though it is not a legal requirement, primary schools take a very dim view of children who rock up at 6 without having been through 3 years of maternelle. Sorry to Labour the point, but on a thread about statistical significance it is a shame to let inaccuracies stand.

There's less grouping by ability going on than there used to be. Partly because it has a number of issues and partly because of the move towards the more mastery styles of assessment and learning that takes a more 'keep up not catch up' sort of approach.

Zooey--my comment about "starting later" was about the legal school starting age, precisely because this is often referenced in discussions about the age at which children start school in the UK, when people say things like, "The UK is very unusual because in most European countries children don't start school until they are 6 or 7." Which is technically true, but kindergartens and their various equivalents in practice tend to play the same kind of role as our Reception and Year 1 do, and in practice the vast majority of children attend them. I think we are actually making the same point, more or less.

In Japan, where I live, Grade 1 starts at age 6, same as France, and technically speaking you do not have to do any schooling at all before this. In practice, the great majority of children attend kindy or daycare for 2-3 years before this.