Science Puzzle
Sharp Shooting the Barn Door
A man fires a rifle at the side of a barn many times, more or less at random. He then walks up, finds a spot where four bullet holes happen to sit close together, paints a target around them, and declares himself a marksman.
A researcher collects data on a hundred variables, notices afterwards that three of them correlate, and publishes a theory explaining those three.
What is the shared error, and why is it invalid?
The Answer
Both specified the target after seeing where the shots landed. This is the Texas sharpshooter fallacy, and the flaw is not that the pattern is imaginary. The four holes really are clustered, and the three variables really do correlate. The flaw is that finding a pattern in data you have already seen is not a test of anything.
The reason is that random data is not featureless. Scatter fifteen shots at a wall and there will almost certainly be a cluster somewhere, purely by chance. Measure a hundred variables and some pairs will correlate, purely by chance. So the existence of a cluster is guaranteed in advance. Pointing at it afterwards tells you nothing about the shooter's aim, because he would have found one whatever he did.
The formal way to say this: a hypothesis has predictive power only when it forbids something in advance. "I will hit that circle" is a real prediction and can fail. "I hit something, and I now declare that to be the circle" cannot fail, and so it carries no information. The move guarantees success by choosing the criterion after the outcome, and anything that cannot fail cannot be evidence.
This is the same underlying error as p-hacking, dressed differently. There, many analyses are run and the one that worked is reported. Here, many outcomes are available and the one that occurred is declared to have been the target. In both cases the number of unreported opportunities is what makes the result meaningless, and in both cases that number is invisible to the reader.
The fix is the same in both settings and is entirely mechanical: state the target before you fire. If a pattern is spotted in existing data, it is not a finding. It is a hypothesis, and it must now be tested against fresh data that has not been looked at, where it is free to fail. Almost all of these dazzling post-hoc patterns quietly evaporate at that point, which is exactly what should happen to them.
The principle: The Texas sharpshooter fallacy. A hypothesis chosen after seeing the data cannot be tested by that data. Random data always contains some pattern, so finding one is guaranteed and therefore uninformative.