I skimmed through the report and skipped some of the daunting polemic about press releases. I tried to ask the questions: what the data do tell, what they don’t tell and why.
The Mastery treatment used in the trial is not an adaptation of the Chinese method by the teachers, as I thought, but a toolkit of materials prepared by a private company. (ups)
This means that the results reflect on the toolkit, rather than on the method in principle and its potential benefits with a better implementation. Those privately developed materials are not necessarily the best and last word in assessing the merits of the Method. They need improving. “The overall conclusion is an indirect criticism of the Toolkit”, states the critique.
The Mastery treatment didn’t involve a different teaching style, just different content, the teachers were not trained for long and it is difficult to ascertain whether they engaged with the method. So if the teaching style was suboptimal for introverted or visual learners, or for less able students, the Mastery method would not have removed the problem. The analysis shows that the effect of the statistical significance of the benefit of the method is smaller than that of a learning style. As I said earlier in the thread, serious research is need with regards to learning style and trials should be controlled for learning style and ability.
-
The data show that with the Mastery system, a limited initial experiment, presumably well controlled, showed better results as illustrated in the graph. This initial basis to start the experiment is good and I would take it for my child.
-
Scaling up the experiment to many schools involved paying a fee to a private company, training the teachers afresh, buying the new to those teachers materials and basically having the teething problems and political polemic inside the experiment, like at Bohunt school. There ought to be some variation and some cultural issues that might pollute the results. I would really start the randomised trial only involving the Mastery trained teachers with several years of experience with the Mastery method and materials. Then you would have sufficient control of control variables in order not to compromise the results…
I understand that the actual Mastery training and materials are from a private company with some American connection. In order to participate, the schools should have paid some £6000 fee and/or the cost of training and materials to that company. There was even a suggestion that this private company would become a permanent feature, resulting in small ongoing cost of £127 per pupil per year, implying privatisation of education and the controversy that comes with it.
So the political context and the resulting resistance to the idea amongst teachers might be quite significant and driven by all sorts of factors nothing to do with the pupils learning. The critique states that “it is not possible to determine the extent to which schools adhered to the prescribed programme. “
Ideally you should have state controlled training and materials to pilot a reform in state education, I would think… Maybe I am misunderstanding, I am sorry, I didn’t try to understand some of the details…
-
So, with all those poorly controlled variables, the results shown overall small improvement of the average attainment of an average student with the Mastery system after a year. The advantage of 1 or 2 months over a school year. Not huge, but nothing wrong with that. I would take it too. 2 months over 13 years is 26 months, which corresponds to the claims that the Chinese students are 3 years ahead…
-
The question is what do those averages mean? The stated purpose of the Mastery system is to not leave anyone behind, to advance in one front. In that case the average attainment of an average student in theory should mean nearly no student performed much worse than that average and they did better than control by 1-2 months… I would take this any time.
-
But yes, unfortunately the data do not decisively prove or disprove that because of questionable materials and poorly controlled implementation. The results reflect poor trial design and political context. So the 95% confidence interval showing the difference between the Mastery group and the control includes zero, which means one could statistically argue there might be no difference overall in pace of the averages. The Mastery system as implemented by the controversial trial did not progress significantly faster than the control, but qualitatively, directionally faster.
Even if the trial did not conclusively prove that, the method that has a potential to improve outcomes for many DC is worth piloting further.
Mainly it shows that the results are encouraging, the private provider aspect and political controversy needs to be taken out of the trial to generate reliable data, that a broader, better controlled trial is needed. What’s not to like?
I just skimmed it, because I feel the big picture is this:
The main advantage of the Mastery method is in not leaving students behind while advancing at an overall similar or faster pace than the current method. I think it is a good thing and worth trying (unless I want to protect the advantage my DC could enjoy in the top set). Of course it needs better design and harder trying from the teachers and removal of private supplier thing. Why not?