Randomised Controlled Trials in English Schools

(47 Posts)
EducationState Sun 12-May-13 18:20:46

Currently, the Department for Education is funding a number of experiments (known as randomised controlled trials (RCTs)) in English schools https://www.gov.uk/government/news/new-randomised-controlled-trials-will-drive-forward-evidence-based-research

Apart from questions regarding whether the £140 million being spent on RCTs is money well-spent, there are other issues concerning the use of RCTs, which parents may not be aware of, which the DfE have perhaps not made as clear as they might, and which by writing this thread, in some small way it is hoped to address.

RCTs are normally conducted in medical research and have proven highly effective, but come with stringent ethical controls, and similar controls will more than likely be in place for those being carried out in English schools.

However, just as in medical research, as such trials are experimental, there is the possibility that an education experiment's instrument/intervention (e.g. new teaching method) will have a negative rather than positive or neutral impact. In medical research, negative impact can be fatal (e.g. the side-effects of a new drug treatment). In education, negative impact is highly unlikely to be fatal, but may, for example, set a child's learning back some time.

This thread, therefore, is meant as both a reminder and a warning. It is a reminder that parents should expect to be notified before their child participates in such an experimental trial and also parents should expect to be given the choice of whether they allow their child to participate. If in the unlikely situation that parents are not notified nor given the choice then this lack of consultation ought to be taken up with the school.

The warning to parents on the other hand is that being experimented on even in educational contexts (e.g. the classroom) could have a negative effect on your child, and this is clearly something that needs bearing in mind.

tiredaftertwo Thu 16-May-13 08:09:40

Good for you Underwater, I hope you can find a way to do both, even if things are very tough at the moment. I think it would be great if more teachers were involved in research.

creamteas Wed 15-May-13 20:58:56

I'm currently supervising a P/T PhD student who is also teaching part time. But she is having to self-fund the PhD. PhD funding is getting harder and harder to get, although there are some funded P/T PhD students, the funding tends to come from their employers (so rarely from the public sector now).

Under I love my job, but being an academic is a lot harder than most people think (a bit like teaching grin). It is brilliant to be able to shape the curriculum and assessment, but we still have disruptive and inattentive students! Our jobs often depend on being able to bring in research grants and you have to get quite think skinned to be able to deal with the constant, and often arbitrary rejections of your research.

UnderwaterBasketWeaving Wed 15-May-13 20:40:16

But I want to be an academic too! I want to do a proper PhD! AND do teaching, cos I quite like it!

tiredaftertwo Tue 14-May-13 20:36:03

I think there are different sorts of collaboration being discussed here.

A few practitioners may be involved in running trials which are based in a university department say, to keep the academics in touch if they haven't got a tame one on hand. And then hundreds or thousands are involved in the actual trial - testing the intervention - this is what most health professionals do in medical ones. There is no way they'd be able to design and run large multi centre trials - it is very highly specialised and skilled in a different way from their day jobs.

I'm with creamteas - but has anyone suggested that it is teachers who run these things? Everything I have seen suggests not.

Which is not to say there are not other types of research that could be combined with teaching. Interventions tested in RCTs have to come from somewhere.......

thecatfromjapan Tue 14-May-13 20:12:09

Me too, Underwaterbasketweaving. I doubt we're alone. That's what makes me think that there will be no way on earth that such an option will be funded.

In fact, so few people on this thread have expressed any sort of negativity of any sort towards this idea, that it makes me think it will be quietly shelved. <cynical face>

UnderwaterBasketWeaving Tue 14-May-13 18:52:33

Can I say now, I'd like to teach part time and do a PhD part time. Including having some statistics and social sciences training. But of course, to do that I'd need the PhD part to be funded and secure.

Surely there are other teachers in the same boat? Or maybe I'm an oddball who still really likes learning and is fascinated by psychology and learning? I can't be the only one?

So do we think that funded, part-time PhD/research positions will become available for teachers to be involved in research AND continue to teach?

creamteas Tue 14-May-13 16:57:26

It is easy to forget that people on the ground are graduates as well and are capable of learning new skills (Doctors, teachers and nurses are all graduates and have above average intelligence.)

In an earlier stage of my career, I taught research methodology to health professionals on Masters courses. The majority of them were unable (or unwilling) to grasp concepts such as a epistemology and ontology. Methods was a core module, but many didn't put any effort into it.

I think they thought that that research design is easy (a bit like the DoE!) and they just couldn't be bothered to engage properly.

ReallyTired Tue 14-May-13 11:35:19

" If you want good research, you need to pay research professionals to do it. "

Then you end up with a situation where you have researtch professionals who have no clue what it is like to teach in a classroom. It is all very well having research professionals stuck in an ivory tower, but schools don't exist in ivory towers.

My impression was that BG wants collaboration between teachers and research professionals. Certainly RCTs have their limitations, but it has to be better than having politicans deciding policy on a whim over breakfast at the tory party conference. RCT can be used alongside other methods of research.

The medical world has had sucess having GPs collaborate with research professionals. It is easy to forget that people on the ground are graduates as well and are capable of learning new skills. (Doctors, teachers and nurses are all graduates and have above average intelligence.)

I am sure that with help teachers can learn to produce good research designs.

noblegiraffe Tue 14-May-13 10:19:41

I agree, creamteas that while RCTs should be properly designed and used carefully, surely the whole notion of having a control group is important for any trial? If you look at the Durham Fish Oil trials I linked to earlier, they were utterly useless from the start because of no control. If people get the idea that they need something to make a comparison against, then that is at least a start.

bunnybing Tue 14-May-13 09:25:34

I think this can only be a good thing. At the moment education policies are changed by successive govs purely for political purposes.

I think the hope with using RCTs is that if something has been proven to work then that policy can remain in place, and not thrown out on a whim.

creamteas Tue 14-May-13 00:05:16

I think some of what Goldacre does is great, and research is important.

But I despair at the lack of understanding of what RCTs can and can't tell us. All methods have strengths and weaknesses and need to be chosen carefully. A move to large numbers of inappropriate RCTs will be just as damaging as relying on poor quality evaluations. I teach research methods up to PhD level including stats, and it is only at doctoral level that most people are able to produce good research designs. If you want good research, you need to pay research professionals to do it.

PGCEs are not designed to provide proper research training, although they should give training in how to interpret evidence. This is actually similar to medicine and nursing BTW. This shouldn't be a problem as, at that level, they are unlikely to be designing research beyond evaluating their own practice.
There is a lot less good educational research that there was for a number of reasons in my opinion. First what research gets commissioned is highly political and the money is decreasing all the time. A lot of this is because the DfE only wants research that will back the position that they are taking anyway (this is a good example).

There are a lot of pressures on getting answers now, even if this will not tell us very much. So trials are often too short to be able to measure anything meaningful. Research is usually put out to tender now and those awarding the contracts often have little research training (hence they can't tell the difference between good and poor quality studies, but cheap is good). And finally, a lot of people doing research (especially those in behaviour intervention studies) still don't understand that correlation is not causation angry.

ReallyTired Mon 13-May-13 22:59:29

creamteas

What do you suggest instead?

At the moment every Tom, Dick and Harry deems themself to be an expert on education because they have been to school.

A PGCE involves no statistics training whatsoever and I would be surprised if a BEd does. I dropped out of PGCE as I found I hated teaching. I got good marks for my essays though! (Quoting bits from relevent papers that support the lecturer's opinons. These papers had little in the way of really rigorous statistics.)

However theory was very different from practice. Many PGCE tutors have not been a classroom for years.

I believe the world of medicine is every bit as complicated as education. Every patient is different just like every pupil is different.

For example establishing the risk factors for childhood leukemia are complex. (Ie. the cluster leukemia cases near Sellafield - this caused by the nuclear power station, population mixing, radioactive rocks, x-rays in pregnancy, diet, drinking water, ante natal care etc) Why did towns like Milton Keynes experience a cluster of childhood leukemia in the 50s?) Was the high number of leukemia cases in gulf war veterans caused by population mixing or depleted uranium? Epidemology uses models to predict the rise in leukemia that would be expected from population mixing or radioactive rocks using compuation modelling.)

Ben Goldacre is a research fellow in Epidemiology. He studies populations for a living. Teachers don't need to be told how to teach, but they may benefit from the knowledge of epidemologists if they want to do rigorous research.

creamteas Mon 13-May-13 22:20:10

I think its reasonable to treat education as a branch of pychology

I don't agree with this at all. Education is a discipline its own right, because it needs to understand much more than psychology. Psychology rarely looks at structures in the way that sociology does for example.

Reallytired Of course multivariate statistics are used, but they still cannot compensate for poorly designed trials (and behavioural interventions are often in this category so would be similar to education RCTs).

Many drug trials are relatively simple in comparison. There is usually a standard treatment to compare to (not so in education) and you can ensure that variations in sample etc can be account for (much harder when the outcomes you are measuring are unclear).

Many behavioural interventions use proxy measures because it is to difficult to measure the thing they want change. For example, common measure for Surestart's success was things like breastfeeding rates. This may or many not be a good thing, but it says very little about if they were successful in reducing social exclusion.

UnderwaterBasketWeaving Mon 13-May-13 20:42:52

"Pychology is both an art and a science. I think its reasonable to treat education as a branch of pychology." - I agree! And you're right about being blinded by passion, I suppose that's partly why a team approach would be better.

Can MN fund a PhD for us?

ReallyTired Mon 13-May-13 20:33:09

"Honestly, teachers wouldn't be trying to study poor practice, they'd only be trying things they truly believed to be positive in influence, and if it became clear that things weren't working they'd stop."

Teachers are human beings and sometimes human beings who passionate about an idea can fail to see flaws. It is very common for any scientist/ researcher to subconciously overlook research findings that conflict with their hypothesis. An ethics commitee will look at a research proposal to see that there are sufficent safeguards and that the experiment is designed well.

"For a start, they can never be 'blind' studies (eg staff involved know which is the control and which the intervention).

Second, you cannot isolate the intervention from other changes (the ethos of school A might change, and thus a positive/negative result may come from this).

Third, they can rarely be direct comparisons, matching intervention and control is extremely difficult. Although if you have enough sites you can try to statistically reduce the chances of this making a difference, but in complex settings, it is still a significant issue. "

There is an area of mathematics called Multivariate Statistics which used in medicine to tease out results. Phara companies employ specialist post graduate statistians to help make sense of data.

I think that teachers doing their own research would help strengthen the teacher's own ablity to think critically.

Pychology is both an art and a science. I think its reasonable to treat education as a branch of pychology.

UnderwaterBasketWeaving Mon 13-May-13 20:22:07

Good points, cream, but these are things that already dog educational research. Attempting the RCT model will at least help with rigour.

As someone up thread mentioned, teachers ideally need to be granted the autonomy and headspace to be able to choose from an armoury of inventive strategies to reach all learners. If this helps to compile a list of things that have been shown to work, which teachers can choose to use, then it's great.

But I do worry about results being hijacked in a dictatorial manner.

creamteas Mon 13-May-13 19:48:50

I'm all for good quality research, but just because it is an RCT does not guarantee it!

RCTs are not necessarily good for measuring the outcome of complex interventions in medicine, let alone other settings

For a start, they can never be 'blind' studies (eg staff involved know which is the control and which the intervention).

Second, you cannot isolate the intervention from other changes (the ethos of school A might change, and thus a positive/negative result may come from this).

Third, they can rarely be direct comparisons, matching intervention and control is extremely difficult. Although if you have enough sites you can try to statistically reduce the chances of this making a difference, but in complex settings, it is still a significant issue.

In fact, the problem with poorly designed RCTs is so great that it is quite usual to have to disregard quite a large proportion when carrying out evidence reviews (sometimes as much as 50%)

tiredaftertwo Mon 13-May-13 19:03:05

These are all good and really interesting points. I hope they will look at what has worked in, say, research in primary care, where they are studying interventions made by people like GPs and mental health nurses, who are not usually academics. RCTs have, I understand, been used in marketing, development, and health education. Those people are not trained quantitative researchers or scientists.

Which is not to say the practitioners should not be involved in the high-levels thinking especially if they are interested and want to be. But it is not necessary I don't think, as long as other practitioners are, at different levels.

The first one they describe looks like all it is is some head teachers being given different data from normal for their comparisons. But, for example, you could do one on whether spelling tests work (I think this one was done a few years ago). The type of data you would need for that - progress through levels, socioeconomic etc etc, will all probably be being collected anyway.

MrsSalvoMontalbano Mon 13-May-13 19:02:51

BG is another media tart who has managed to do a Jamie Oliver on the government - I doubt he was picked - probably just bowled up and flashed his meeja credentials. Stuff will happen, there will be headlines, he gets a bit more limelight, it will be quitely ignored and and like everything, that too shall pass.

noblegiraffe Mon 13-May-13 18:59:35

I'm not sure Goldacre's full report has been linked to yet, it's long but well worth a full read as it answers many questions
www.badscience.net/2013/03/heres-my-paper-on-evidence-and-teaching-for-the-education-minister/

On the ethical point, it says
Firstly, people sometimes worry that it is unethical to randomly assign children to one educational intervention or another. Often this is driven by an implicit belief that a new or expensive intervention is always necessarily better. When people believe this, they also worry that it’s wrong to deprive people of the new intervention. It’s important to be clear, before we get to the detail, that a trial doesn’t necessarily involve depriving people of anything, since we can often run a trial where people are randomly assigned to receive the new intervention now, or after a six month wait. But there is a more important reason why trials are ethically acceptable: in reality, before we do a trial, we generally have no idea which of two interventions is best. Furthermore, new things that many people believe in can sometimes turn out, in reality, to be very harmful.

As for teaching being an art rather than a science, while that is true for certain aspects of classroom practice, there are many many things which would be suitable for investigation. E.g. Does teaching reading with phonics improve English reading results? Does banning calculators in primary school improve maths results?

UnderwaterBasketWeaving Mon 13-May-13 18:44:02

In fact, ReallyTired you've made a bunch of really good points.

Teachers (and I mean teachers, not educational researchers who don't teach and have therefore forgotten/don't know how things translate into a real classroom situation) do need to be supported, do need to be given the time and freedom to be involved in this. And, most definitely as part of a team.

NobleGiraffe, you mentioned teachers taking on extra work to do this. I think if the government want serious and good research, they are going to have to support schools to allow teachers to remain on full pay but to have part time teaching hours with the rest of the time collaborating with universities. To do this well, it would be impossible to do it alongside a full teaching load (which is impossible in its own right most of the time!)

UnderwaterBasketWeaving Mon 13-May-13 18:37:23

ReallyTired "Rather than asking permission, prehaps experiments should need ethical approval."

Honestly, teachers wouldn't be trying to study poor practice, they'd only be trying things they truly believed to be positive in influence, and if it became clear that things weren't working they'd stop.

I imagine ethical approval would be required initially, but the biggest issues would be in ensuring that SEN/G&T students would be sufficiently supported (although anything good enough for study by RCT should be easily differentiated and to the benefit of all learners, otherwise what's the point?!)

UnderwaterBasketWeaving Mon 13-May-13 18:32:36

Ben Goldacre is very well qualified to promote RCT in educational research.

He has been campaigning for many years to change the publishing policies of pharmacological research, as at the moment drug companies are not required to publish studies that show their drug to be dangerous or ineffective. Instead the studies are just halted and swept under the carpet. Meaning the ONLY published work is in support of the drug companies.

In terms of educational research, this is just formalising what teachers do anyway. They try different teaching techniques with different groups at different times as a matter of course. What works for a very able group, just won't work for a more challenged group, so they try different tactics to find something that works. Also, teachers may change techniques to keep the classroom vibrant, to keep challenging students rather than letting them fall into routine that might become tired and unstimulating.

Until now, educational research has been carried out as "Action Research". Educational research is like the even-more-rubbish version of social science research.

Introducing this kind of rigour, to assess the effect of things that teachers do normally, is a good thing (unless you happen to be the teacher trying to write up research on top of a full timetable!).

thecatfromjapan Mon 13-May-13 18:20:41

Sorry.

I'm probably just a bit defensive. I've been away for a bit and ... I don't know. It all seems a lot rougher than I remember.

Starlight - that's a very interesting break from the "BG is great" consensus I usually hear - I have a few rather forceful scientist friends who think he is the leader of the righteous. grin Refreshing, actually, to read you saying that.

Trills - Ben Goldacre is the author of this report we're discussing. He's a science-based journalist and a best-selling author. He writes for the Guardian.

Now, here is what I want to know: who is Education State?

tiredaftertwo Mon 13-May-13 17:43:23

Umm, I think the questions were brisk and precise, not aggressive. Thecat, I thought your first post was clear, and made sense. The OP did, however, issue a clear warning and a call for parental consent to be required and people are discussing the implications of that. That's fair enough, surely?

I have seen the behaviour you describe on other threads - and share your frustration - but I do not see it here, and I think there have been some good descriptions of the way complex multidisciplinary research can work.

Join the discussion

Join the discussion

Registering is free, easy, and means you can join in the discussion, get discounts, win prizes and lots more.

Register now