Randomised Controlled Trials in English Schools(47 Posts)
Currently, the Department for Education is funding a number of experiments (known as randomised controlled trials (RCTs)) in English schools https://www.gov.uk/government/news/new-randomised-controlled-trials-will-drive-forward-evidence-based-research
Apart from questions regarding whether the £140 million being spent on RCTs is money well-spent, there are other issues concerning the use of RCTs, which parents may not be aware of, which the DfE have perhaps not made as clear as they might, and which by writing this thread, in some small way it is hoped to address.
RCTs are normally conducted in medical research and have proven highly effective, but come with stringent ethical controls, and similar controls will more than likely be in place for those being carried out in English schools.
However, just as in medical research, as such trials are experimental, there is the possibility that an education experiment's instrument/intervention (e.g. new teaching method) will have a negative rather than positive or neutral impact. In medical research, negative impact can be fatal (e.g. the side-effects of a new drug treatment). In education, negative impact is highly unlikely to be fatal, but may, for example, set a child's learning back some time.
This thread, therefore, is meant as both a reminder and a warning. It is a reminder that parents should expect to be notified before their child participates in such an experimental trial and also parents should expect to be given the choice of whether they allow their child to participate. If in the unlikely situation that parents are not notified nor given the choice then this lack of consultation ought to be taken up with the school.
The warning to parents on the other hand is that being experimented on even in educational contexts (e.g. the classroom) could have a negative effect on your child, and this is clearly something that needs bearing in mind.
That's fantastic news, I'm really pleased that there will be a decent evidence base for what is passed off as best practice in schools.
At the moment kids are experimented on anyway with endless new initiatives. At least now we might get some idea of whether these initiatives are working.
I hope it does have the effect you see, noblegiraffe.
I find the constant, unregulated experimentation totally annoying too, and I'm just a parent. I fondly hope that my children's teachers will buffer them from the worst of the wild changes. <fret>
Anyway, let's just hope it all goes as Mr Goldacre hopes.
Was he an odd choice to write this report? Is he a proper expert in cross-disciplinary research? I only know him as a quasi-celebrity science journalist. Not that that necessarily means this is a bad idea. Stamps were "invented" by a non-expert in anything, weren't they? They were a good idea. And I do like a lot of what he writes.
By the way, what if you want to opt your child out of one of these trials? Would the LEA have to place your child in another school; could you choose that school; would they pay for transport if it is a long way away?
I don't think its realistic to ask parents' permission before doing an educational experiment on their child. Teachers need a certain level of automony and should not be required to ask parents for permission every time they make a decision. Rather than asking permission, prehaps experiments should need ethical approval.
The way I read e release is that they are not imposing a teaching method as part f the RCT. Rather that they are matching cohorts of schools by intake, then monitoring the teaching and other interventions (that the schools are selecting anyhow) then comparing outcomes. There would then be a collaborative effort between the participating schools and researchers to establish, on the basis of evidence, which methods and interventions lead to the best outcomes. And they can then be encouraged for general use on the basis of evidence (rather than just imposing something that someone somewhere thinks is a good idea).
I think RCTs in education are a great idea. Having a strong evidence base for what is more or less effective can only be a good thing.
It has benefited our medical health so why not try it on other aspects of development. As Noblegiraffe says - at least then people will be able and expected to prove / justify new initiatives or explain why they are doing things a certain way. Very happy for DD to be involved (and surely no different to the present where one teacher, school or LA takes one approach while another does it differently).
I think Ben Goldacre was / is originally a doctor, which might be why he was chosen.
I know he's a trained medic. but there are a lot of other people who are extremely qualified in this area - Professors of medical ethics; specialists in the development of randomised trials; professional educationalists; professionals involved in the development of public policy; etc. and they weren't asked.
As I said, I read BG's paper and I liked it. I was particularly interested in his suggestions about teachers being at the forefront of developing these trials and in his idea that career progression be linked to undertaking research.
I suppose I am just worried that it is all just a bit of "puff" for some current experiments that are underway at the moment and will just disappear. I think what noblegiraffe said in her first post is pretty much where I stand on this: the massive, sweeping changes that have been inflicted - they really need to stop.
I guess I worry that they have chosen BG for headlines, and precisely because he lacks professional clout in this field there will be no comeback when it all quietly disappears.
I really did think BG's suggestions were very interesting.
Education is strongly linked to child pychology. Child pychology is linked to medicine. If RCT trials have dramatically improved medicine then they may well improve education. I don't think we should dismiss Goldburg's suggestion just because he is a doctor.
Finding out what inteventions work is a job for a skill statistican rather than a teacher. Clearly teachers need to work closely with top statisticans to use each other's skill sets to find out what works.
Our year 6 children are all part of big experiment this week when they sit the grammar paper. Some of our youngest children (year 1) are going to be doing phonics test in a few weeks time. All these changes have been done with no randomised trials.
Ben Goldacre will have been picked (if indeed he was picked, rather, I suspect that he simply browbeat the government into taking notice of his suggestions and this is all his idea) because he has been banging on about this stuff for years.
Here are his many articles on the appallingly designed Durham Fish Oil trials
And his many articles on Brain Gym
Remember, this rubbish has been going on in schools with real children.
He's a big fan of evidence-based policy in politics too, not just education.
Not sure why Ben Goldacre is considered the best person to front this, but moving to evidence based intervention can only be a good thing. But the use of the term RCT is a bit misleading. The first study described sounds like more of a longditudanal study than a RCT.
Anyway, just to balance a previous posters comment that RCT's can result in fatal side effects. It is fairly rare that this occurs and well designed RCT's are usually preceeded by a pilot small scale study and have to go through stringent regulatory and ethics approval - unlike the sort of experimentation we frequently see in education
I suppose that Ben Goldacre has been picked for his enthusiam and passion for the use RCT in all different areas of life.
I feel its important that teachers are at the forefront of research, but someone who has a degree in early years may need help with complex post graduate statistics. Everyone has different talents and a good RCT needs a range of people with different talents.
Many years ago I was part of a research team of hundreds of scientists and mathematicans world wide who did research into the affects of depleted uranium on soldiers. No one had the knowledge to do the research on their own.
Maybe we can have better analysis of existing interventions as well as experiments.
Good research is collabative. Teachers don't have a strong research tradition and prehaps help from other professions to make teaching a research based profession.
How else do you suppose we go about finding out what works and what does not?
I don't think he was an odd choice to front this campaign, but then I have seen him talk. He is an articulate and enthusiastic campaigner for evidence-based decision making and a consummate dispeller of pseudo-science and bullshit claims. He has written before about the poor evidence base for certain school initiatives (also see his interesting discussion on the evidence about whether exams are getting easier) so he hasn't solely concentrated on health in the past.
He is also an incredibly engaging speaker and so may actually (as opposed to some dry academic) be able to convince teachers to take on extra work and conduct research in their own classrooms.
I don't think it really matters that he doesn't work in education because this isn't about education really, it's about improving the evidence base for initiatives in education.
I think concerns about experimentation are unfounded.
Education practice and policy changes all the time, often with a shaky evidence base (yes, Mr Gove, I'm looking at you, stop fiddling with public exams in the back row, and listen to me when I'm talking).
The changes are often not controlled, outcomes may not be defined beforehand or carefully monitored, and so on. One of the many advantages of this way is that negative impacts will be likely to be picked up sooner and if serious the trial can be abandoned (as happens in medical research).
It is good news IMO - if teachers can be persuaded to get on board.
So, between experimenting and gradually implementing changes proven to work and rushing in changes that a random politician thinks is best as it represents his schooling - the RCTs are the thing you are worried about? Really?
Are all these slightly aggressive questions directed to me?
If you read my posts, it's clear that I find the idea interesting, and that I have actually read Ben Goldman's paper (before this thread was started; because I was genuinely interested in the idea; because I generally like Ben Goldman's work) - I don't think many of you have, and yet I'm supposed to be a straw man against which you are constructing some hallucinatory argument? Odd.
If you'd read his paper, you would know that, yes, Reallytired, Ben Goldman envisions these trials to be team efforts in the main, with teachers mainly involved in shaping the studies, putting them into practice, and identifying areas of possible research. Though obviously, if you are a statistician and a teacher, you'll be off to a head start in any number-crunching. But, I see your response there as a genuine, interested response. And it was nice to read it.
I'm depressed, frankly, by the snippiness, and yes, the stupidity of some of the responses to my questions. There has been a tendency to read anything posted on mn increasingly in the most cynical, aggressive and mean-spirited fashion. It doesn't enable the spread and facilitating of ideas and creativity - it actively inhibits it. That is what I mean by stupidity. You are not stupid women. I recognise you from other threads. So why are you acting here in a fashion to put a full stop to thought, rather than promoting a conversation?
I can;t help but feel my only "crime" was to ask why Ben Goldman was chosen. I think that's a valid question. I very, very sincerely am interested in this idea. Everyone on this thread has pointed out that the current situation really shouldn't prevail. I do wonder if this might be the equivalent in education of Labour giving the Bank of England its independence. It's fascinating. So it is worth considering exactly how seriously the government intend to take this.
I'll repeat: there is no-one on this thread arguing that this is a bad idea. To pretend otherwise is, frankly, the equivalent of picking a rather drunken fight at your mate's birthday party. It's not a good look.
Seriously, when the fuck did mn change from a place you could have collaborative, constructive conversations - based on an assumption that the majority of you understand and accept the rules of civilised communication - to a multi-topical argument - with the rules of the bunfight?
Sorry if I sound pissed off - I am. This is not AIBU.
thecatfromjapan are you sure you posted that on the right thread? Who do you think is having a go at you? And who is Ben Goldman?
Cat, I think the slightly aggressive questions are directed at the OP, not you!
Kids are experimented on all the time in schools, by latest daft government initiatives, to HT's 'turnaround' styles to whims of the class teachers.
RCT at least regulate this and build up a bank of knowledge about effectiveness and I would hope, give the teaching participants a reminder/education/insight into EBP.
FWIW, I think BG is a patronising arse.
And imo teachers prefer to think of teaching as an art rather than a science so are (in a general sense) unlikely to get on board.
Umm, I think the questions were brisk and precise, not aggressive. Thecat, I thought your first post was clear, and made sense. The OP did, however, issue a clear warning and a call for parental consent to be required and people are discussing the implications of that. That's fair enough, surely?
I have seen the behaviour you describe on other threads - and share your frustration - but I do not see it here, and I think there have been some good descriptions of the way complex multidisciplinary research can work.
I'm probably just a bit defensive. I've been away for a bit and ... I don't know. It all seems a lot rougher than I remember.
Starlight - that's a very interesting break from the "BG is great" consensus I usually hear - I have a few rather forceful scientist friends who think he is the leader of the righteous. Refreshing, actually, to read you saying that.
Trills - Ben Goldacre is the author of this report we're discussing. He's a science-based journalist and a best-selling author. He writes for the Guardian.
Now, here is what I want to know: who is Education State?
Ben Goldacre is very well qualified to promote RCT in educational research.
He has been campaigning for many years to change the publishing policies of pharmacological research, as at the moment drug companies are not required to publish studies that show their drug to be dangerous or ineffective. Instead the studies are just halted and swept under the carpet. Meaning the ONLY published work is in support of the drug companies.
In terms of educational research, this is just formalising what teachers do anyway. They try different teaching techniques with different groups at different times as a matter of course. What works for a very able group, just won't work for a more challenged group, so they try different tactics to find something that works. Also, teachers may change techniques to keep the classroom vibrant, to keep challenging students rather than letting them fall into routine that might become tired and unstimulating.
Until now, educational research has been carried out as "Action Research". Educational research is like the even-more-rubbish version of social science research.
Introducing this kind of rigour, to assess the effect of things that teachers do normally, is a good thing (unless you happen to be the teacher trying to write up research on top of a full timetable!).
ReallyTired "Rather than asking permission, prehaps experiments should need ethical approval."
Honestly, teachers wouldn't be trying to study poor practice, they'd only be trying things they truly believed to be positive in influence, and if it became clear that things weren't working they'd stop.
I imagine ethical approval would be required initially, but the biggest issues would be in ensuring that SEN/G&T students would be sufficiently supported (although anything good enough for study by RCT should be easily differentiated and to the benefit of all learners, otherwise what's the point?!)
In fact, ReallyTired you've made a bunch of really good points.
Teachers (and I mean teachers, not educational researchers who don't teach and have therefore forgotten/don't know how things translate into a real classroom situation) do need to be supported, do need to be given the time and freedom to be involved in this. And, most definitely as part of a team.
NobleGiraffe, you mentioned teachers taking on extra work to do this. I think if the government want serious and good research, they are going to have to support schools to allow teachers to remain on full pay but to have part time teaching hours with the rest of the time collaborating with universities. To do this well, it would be impossible to do it alongside a full teaching load (which is impossible in its own right most of the time!)
I'm not sure Goldacre's full report has been linked to yet, it's long but well worth a full read as it answers many questions
On the ethical point, it says
Firstly, people sometimes worry that it is unethical to randomly assign children to one educational intervention or another. Often this is driven by an implicit belief that a new or expensive intervention is always necessarily better. When people believe this, they also worry that its wrong to deprive people of the new intervention. Its important to be clear, before we get to the detail, that a trial doesnt necessarily involve depriving people of anything, since we can often run a trial where people are randomly assigned to receive the new intervention now, or after a six month wait. But there is a more important reason why trials are ethically acceptable: in reality, before we do a trial, we generally have no idea which of two interventions is best. Furthermore, new things that many people believe in can sometimes turn out, in reality, to be very harmful.
As for teaching being an art rather than a science, while that is true for certain aspects of classroom practice, there are many many things which would be suitable for investigation. E.g. Does teaching reading with phonics improve English reading results? Does banning calculators in primary school improve maths results?
BG is another media tart who has managed to do a Jamie Oliver on the government - I doubt he was picked - probably just bowled up and flashed his meeja credentials. Stuff will happen, there will be headlines, he gets a bit more limelight, it will be quitely ignored and and like everything, that too shall pass.
These are all good and really interesting points. I hope they will look at what has worked in, say, research in primary care, where they are studying interventions made by people like GPs and mental health nurses, who are not usually academics. RCTs have, I understand, been used in marketing, development, and health education. Those people are not trained quantitative researchers or scientists.
Which is not to say the practitioners should not be involved in the high-levels thinking especially if they are interested and want to be. But it is not necessary I don't think, as long as other practitioners are, at different levels.
The first one they describe looks like all it is is some head teachers being given different data from normal for their comparisons. But, for example, you could do one on whether spelling tests work (I think this one was done a few years ago). The type of data you would need for that - progress through levels, socioeconomic etc etc, will all probably be being collected anyway.
I'm all for good quality research, but just because it is an RCT does not guarantee it!
RCTs are not necessarily good for measuring the outcome of complex interventions in medicine, let alone other settings
For a start, they can never be 'blind' studies (eg staff involved know which is the control and which the intervention).
Second, you cannot isolate the intervention from other changes (the ethos of school A might change, and thus a positive/negative result may come from this).
Third, they can rarely be direct comparisons, matching intervention and control is extremely difficult. Although if you have enough sites you can try to statistically reduce the chances of this making a difference, but in complex settings, it is still a significant issue.
In fact, the problem with poorly designed RCTs is so great that it is quite usual to have to disregard quite a large proportion when carrying out evidence reviews (sometimes as much as 50%)
Good points, cream, but these are things that already dog educational research. Attempting the RCT model will at least help with rigour.
As someone up thread mentioned, teachers ideally need to be granted the autonomy and headspace to be able to choose from an armoury of inventive strategies to reach all learners. If this helps to compile a list of things that have been shown to work, which teachers can choose to use, then it's great.
But I do worry about results being hijacked in a dictatorial manner.
"Honestly, teachers wouldn't be trying to study poor practice, they'd only be trying things they truly believed to be positive in influence, and if it became clear that things weren't working they'd stop."
Teachers are human beings and sometimes human beings who passionate about an idea can fail to see flaws. It is very common for any scientist/ researcher to subconciously overlook research findings that conflict with their hypothesis. An ethics commitee will look at a research proposal to see that there are sufficent safeguards and that the experiment is designed well.
"For a start, they can never be 'blind' studies (eg staff involved know which is the control and which the intervention).
Second, you cannot isolate the intervention from other changes (the ethos of school A might change, and thus a positive/negative result may come from this).
Third, they can rarely be direct comparisons, matching intervention and control is extremely difficult. Although if you have enough sites you can try to statistically reduce the chances of this making a difference, but in complex settings, it is still a significant issue. "
There is an area of mathematics called Multivariate Statistics which used in medicine to tease out results. Phara companies employ specialist post graduate statistians to help make sense of data.
I think that teachers doing their own research would help strengthen the teacher's own ablity to think critically.
Pychology is both an art and a science. I think its reasonable to treat education as a branch of pychology.
"Pychology is both an art and a science. I think its reasonable to treat education as a branch of pychology." - I agree! And you're right about being blinded by passion, I suppose that's partly why a team approach would be better.
Can MN fund a PhD for us?
I think its reasonable to treat education as a branch of pychology
I don't agree with this at all. Education is a discipline its own right, because it needs to understand much more than psychology. Psychology rarely looks at structures in the way that sociology does for example.
Reallytired Of course multivariate statistics are used, but they still cannot compensate for poorly designed trials (and behavioural interventions are often in this category so would be similar to education RCTs).
Many drug trials are relatively simple in comparison. There is usually a standard treatment to compare to (not so in education) and you can ensure that variations in sample etc can be account for (much harder when the outcomes you are measuring are unclear).
Many behavioural interventions use proxy measures because it is to difficult to measure the thing they want change. For example, common measure for Surestart's success was things like breastfeeding rates. This may or many not be a good thing, but it says very little about if they were successful in reducing social exclusion.
What do you suggest instead?
At the moment every Tom, Dick and Harry deems themself to be an expert on education because they have been to school.
A PGCE involves no statistics training whatsoever and I would be surprised if a BEd does. I dropped out of PGCE as I found I hated teaching. I got good marks for my essays though! (Quoting bits from relevent papers that support the lecturer's opinons. These papers had little in the way of really rigorous statistics.)
However theory was very different from practice. Many PGCE tutors have not been a classroom for years.
I believe the world of medicine is every bit as complicated as education. Every patient is different just like every pupil is different.
For example establishing the risk factors for childhood leukemia are complex. (Ie. the cluster leukemia cases near Sellafield - this caused by the nuclear power station, population mixing, radioactive rocks, x-rays in pregnancy, diet, drinking water, ante natal care etc) Why did towns like Milton Keynes experience a cluster of childhood leukemia in the 50s?) Was the high number of leukemia cases in gulf war veterans caused by population mixing or depleted uranium? Epidemology uses models to predict the rise in leukemia that would be expected from population mixing or radioactive rocks using compuation modelling.)
Ben Goldacre is a research fellow in Epidemiology. He studies populations for a living. Teachers don't need to be told how to teach, but they may benefit from the knowledge of epidemologists if they want to do rigorous research.
I think some of what Goldacre does is great, and research is important.
But I despair at the lack of understanding of what RCTs can and can't tell us. All methods have strengths and weaknesses and need to be chosen carefully. A move to large numbers of inappropriate RCTs will be just as damaging as relying on poor quality evaluations. I teach research methods up to PhD level including stats, and it is only at doctoral level that most people are able to produce good research designs. If you want good research, you need to pay research professionals to do it.
PGCEs are not designed to provide proper research training, although they should give training in how to interpret evidence. This is actually similar to medicine and nursing BTW. This shouldn't be a problem as, at that level, they are unlikely to be designing research beyond evaluating their own practice.
There is a lot less good educational research that there was for a number of reasons in my opinion. First what research gets commissioned is highly political and the money is decreasing all the time. A lot of this is because the DfE only wants research that will back the position that they are taking anyway (this is a good example).
There are a lot of pressures on getting answers now, even if this will not tell us very much. So trials are often too short to be able to measure anything meaningful. Research is usually put out to tender now and those awarding the contracts often have little research training (hence they can't tell the difference between good and poor quality studies, but cheap is good). And finally, a lot of people doing research (especially those in behaviour intervention studies) still don't understand that correlation is not causation .
I think this can only be a good thing. At the moment education policies are changed by successive govs purely for political purposes.
I think the hope with using RCTs is that if something has been proven to work then that policy can remain in place, and not thrown out on a whim.
I agree, creamteas that while RCTs should be properly designed and used carefully, surely the whole notion of having a control group is important for any trial? If you look at the Durham Fish Oil trials I linked to earlier, they were utterly useless from the start because of no control. If people get the idea that they need something to make a comparison against, then that is at least a start.
" If you want good research, you need to pay research professionals to do it. "
Then you end up with a situation where you have researtch professionals who have no clue what it is like to teach in a classroom. It is all very well having research professionals stuck in an ivory tower, but schools don't exist in ivory towers.
My impression was that BG wants collaboration between teachers and research professionals. Certainly RCTs have their limitations, but it has to be better than having politicans deciding policy on a whim over breakfast at the tory party conference. RCT can be used alongside other methods of research.
The medical world has had sucess having GPs collaborate with research professionals. It is easy to forget that people on the ground are graduates as well and are capable of learning new skills. (Doctors, teachers and nurses are all graduates and have above average intelligence.)
I am sure that with help teachers can learn to produce good research designs.
It is easy to forget that people on the ground are graduates as well and are capable of learning new skills (Doctors, teachers and nurses are all graduates and have above average intelligence.)
In an earlier stage of my career, I taught research methodology to health professionals on Masters courses. The majority of them were unable (or unwilling) to grasp concepts such as a epistemology and ontology. Methods was a core module, but many didn't put any effort into it.
I think they thought that that research design is easy (a bit like the DoE!) and they just couldn't be bothered to engage properly.
Can I say now, I'd like to teach part time and do a PhD part time. Including having some statistics and social sciences training. But of course, to do that I'd need the PhD part to be funded and secure.
Surely there are other teachers in the same boat? Or maybe I'm an oddball who still really likes learning and is fascinated by psychology and learning? I can't be the only one?
So do we think that funded, part-time PhD/research positions will become available for teachers to be involved in research AND continue to teach?
Me too, Underwaterbasketweaving. I doubt we're alone. That's what makes me think that there will be no way on earth that such an option will be funded.
In fact, so few people on this thread have expressed any sort of negativity of any sort towards this idea, that it makes me think it will be quietly shelved. <cynical face>
I think there are different sorts of collaboration being discussed here.
A few practitioners may be involved in running trials which are based in a university department say, to keep the academics in touch if they haven't got a tame one on hand. And then hundreds or thousands are involved in the actual trial - testing the intervention - this is what most health professionals do in medical ones. There is no way they'd be able to design and run large multi centre trials - it is very highly specialised and skilled in a different way from their day jobs.
I'm with creamteas - but has anyone suggested that it is teachers who run these things? Everything I have seen suggests not.
Which is not to say there are not other types of research that could be combined with teaching. Interventions tested in RCTs have to come from somewhere.......
But I want to be an academic too! I want to do a proper PhD! AND do teaching, cos I quite like it!
I'm currently supervising a P/T PhD student who is also teaching part time. But she is having to self-fund the PhD. PhD funding is getting harder and harder to get, although there are some funded P/T PhD students, the funding tends to come from their employers (so rarely from the public sector now).
Under I love my job, but being an academic is a lot harder than most people think (a bit like teaching ). It is brilliant to be able to shape the curriculum and assessment, but we still have disruptive and inattentive students! Our jobs often depend on being able to bring in research grants and you have to get quite think skinned to be able to deal with the constant, and often arbitrary rejections of your research.
Good for you Underwater, I hope you can find a way to do both, even if things are very tough at the moment. I think it would be great if more teachers were involved in research.
Join the discussion
Registering is free, easy, and means you can join in the discussion, get discounts, win prizes and lots more.Register now
Already registered with Mumsnet? Log in to leave your comment or alternatively, sign in with Facebook or Google.
Please login first.