Science Puzzle
The Drawer of Failed Studies
A drug has no effect whatsoever. One hundred research teams around the world test it independently. By chance, about five of them get a positive-looking result at the usual significance threshold; the other ninety-five find nothing.
The five positive teams write up their exciting findings and get published. The ninety-five who found nothing conclude their study was boring, or their paper is rejected as uninteresting, and it goes in a drawer.
A doctor now reads every published paper on this drug. Every single one is honest and correctly analysed. What will she conclude, and what has gone wrong?
The Answer
She will conclude the drug works, and she will be reading the evidence correctly. From where she sits, the literature contains five independent studies finding an effect and zero finding none. That looks like a solid, replicated result. It is a mirage.
Nobody lied. Every published paper is honest and correctly analysed. The distortion happened in what never got written up. This is the file drawer problem, or publication bias, and it is one of the most serious structural weaknesses in science, precisely because it cannot be detected by examining any individual paper. The flaw is invisible in the published record by construction: the missing studies are missing.
It is driven by ordinary incentives rather than fraud. Journals prefer positive findings; a paper reporting "we looked and found nothing" is hard to place. Researchers know this, and a null result does little for a career, so it is quietly abandoned. Multiply that across a field and the literature drifts away from reality while every component of it remains truthful.
It also makes p-hacking far more dangerous. If one team runs twenty analyses and publishes the one that worked, that is one distorted paper. If a hundred teams each publish only their successes, the entire field is distorted, and the meta-analyses built on top of that literature will faithfully summarise the distortion.
The remedy that actually bites is pre-registration: researchers publicly declare their hypothesis and analysis plan before collecting data, and the study is registered whatever the outcome. Registered reports go further, with journals accepting the paper on the strength of the method, before the results exist. That makes a null result publishable by design and closes the drawer. Where pre-registration has been introduced, the proportion of positive findings has dropped sharply, which tells you how much was being hidden.
The principle: Publication bias and the file drawer problem. When null results go unpublished, the visible literature over-represents positive findings even though every individual paper is honest, and the flaw is undetectable from within the literature itself.