Science Puzzle
The Worst Schools Improve
A region gives an expensive new teaching programme to the ten worst-performing schools. The next year, those schools improve. The ten best-performing schools, which received nothing, decline slightly.
The minister announces that the programme works and should be rolled out nationally. What is the flaw?
The Answer
The schools were chosen precisely because they scored worst. Any score is part genuine ability and part luck: a difficult cohort, an outbreak of flu on exam week, a couple of unlucky question choices. The schools at the very bottom are, disproportionately, the ones that had a bad year on both counts.
Next year the genuine ability is still there, but the bad luck is not repeated, because luck does not persist. So they drift upward. The schools at the very top were disproportionately the ones having a lucky year, and they drift downward. This is regression to the mean, and it happens whenever you select on an extreme and then measure again. It requires no cause at all.
The cruel consequence is that regression to the mean will systematically flatter any intervention aimed at the worst performers and systematically damn any intervention aimed at the best. It fabricates an apparent effect out of nothing, in the exact direction the minister was hoping for.
This is not a curiosity. It is one of the most common ways that useless interventions acquire an evidence base: hospitals that target their worst wards, coaches who bench a player after one bad game, businesses that overhaul their weakest branch. The subsequent improvement is real, and the credit is misplaced.
The fix is the same one as always: a control group. Take the twenty worst schools, give the programme to a randomly chosen ten, and give nothing to the other ten. Both sets will regress upward. If the treated ten rise further than the untreated ten, that difference is the real effect of the programme, with the regression subtracted out.
The principle: Regression to the mean. Groups selected for extreme scores move toward the average on remeasurement without any intervention, which fabricates an apparent effect for anything targeted at the worst performers.