Science Puzzle
The Keys Under the Streetlight
Late one night, a passer-by finds a man on his hands and knees under a streetlamp, searching the pavement. "I've lost my keys," he explains. The passer-by helps him look for a few minutes, finds nothing and asks: "Are you sure you dropped them here?" "No," the man replies, "I dropped them over there, by my car. But this is where the light is."
It's an old joke, but scientists tell it about themselves. What mistake in research does it describe?
The Answer
Looking for answers where it's easiest to look, rather than where the answer is likely to be. It's known as the streetlight effect.
Research is expensive and slow, so there's a constant temptation to study whatever is cheapest and easiest to measure. The trouble is that the easy place isn't necessarily the right place. Search under the lamp for long enough and you can produce lots of careful results, while the keys sit untouched in the dark.
It happens in real science. For decades many medical studies were carried out mostly on men, partly because they were simpler to recruit and study, and some findings turned out not to apply equally to women. A great deal of psychology has been studied mainly using university students in a few wealthy countries, because they're conveniently on hand, yet conclusions are often stated as if they were true of everyone. For centuries astronomers could only look at visible light; once telescopes could pick up radio waves and X-rays, they found whole classes of objects, such as pulsars, that had been there all along.
The cure isn't to stop using the light. It's to keep asking whether the lit-up patch is where the answer is likely to be, and to say clearly what you couldn't look at.
The principle: The streetlight effect. Searching only where it's easy to look, rather than where the answer is likely to be, can produce plenty of results that miss the thing that matters.