Science Puzzle
The Poll of Two Million People
In 1936 the Literary Digest, a popular American magazine, ran one of the largest opinion polls ever attempted. It posted mock ballots to more than 10 million people, taking their names from telephone directories, car registration records and its own list of subscribers. About 2.4 million ballots came back with a clear result: Alf Landon would beat President Franklin D. Roosevelt by 57% to 43%.
On election day Roosevelt won by a landslide. How could a poll of 2.4 million people get it so badly wrong?
The Answer
The sample was huge, but it wasn't a fair picture of the voters; no amount of size can fix that. Roosevelt took about 61% of the vote and won 46 of the 48 states.
The first problem was where the names came from. In 1936 America was deep in the Great Depression. Owning a telephone or a car, or paying for a magazine, meant you were better off than most, and better-off voters leaned towards Landon. Millions of poorer voters, who strongly backed Roosevelt, were barely on the lists at all.
The second problem was who chose to reply. Fewer than a quarter of the people asked sent their ballots back, and people who bother to answer a survey can differ from those who don't. Later studies of the poll suggest this did at least as much damage as the mailing lists.
A bigger sample only shrinks the random wobble in a result. It does nothing about a sample that leans one way, so the Digest's poll gave a very precise answer to the wrong question. In the same election a young pollster called George Gallup used a far smaller sample chosen to match the make-up of the voting population and correctly predicted Roosevelt's win. The Literary Digest closed down soon afterwards.
The principle: Sampling bias. A huge sample drawn from the wrong people gives a precise but wrong answer, because how a sample is chosen matters far more than how big it is.