Science Puzzle
Twenty Jellybean Colours
Researchers test whether jellybeans cause acne. They find nothing. So they test each colour separately: purple, brown, pink, blue, and so on through twenty colours.
Nineteen colours show no link. Green jellybeans show a link at the standard 95 percent confidence level. The newspapers run the headline. Every individual statistical test was performed correctly.
What is wrong?
The Answer
Twenty tests at 95 percent confidence is the whole problem. The 95 percent threshold means that a result this strong would appear by pure chance about 1 time in 20 even when there is no real effect whatsoever.
So if you run 20 independent tests on something that does nothing, you should expect, on average, about one of them to come back "significant" anyway. Finding exactly one significant colour out of twenty is not evidence of an effect. It is precisely the result you predict if jellybeans are entirely inert.
This is the multiple comparisons problem. It is not a failure of arithmetic; every individual test was computed correctly. It is a failure of interpretation, because the significance threshold was designed for a single pre-planned test, not for the best of twenty.
The manipulation becomes far more serious when it is hidden. If the paper reports only the green result and quietly omits the nineteen failures, the reader cannot possibly assess it. Running many analyses and publishing only the flattering one is known as p-hacking, and it is one of the main reasons published findings so often fail to replicate.
The defences are straightforward and are the mark of honest work: state in advance which single hypothesis you are testing, adjust the threshold when you run many tests, report every test you ran and not just the ones that worked, and treat any surprise finding from a fishing expedition as a hypothesis to be tested afresh on new data rather than as a discovery.
The principle: Multiple comparisons and p-hacking. A 95% threshold produces roughly one false positive per 20 tests. Running many tests and reporting only the significant one manufactures findings from noise.