Statistics & Data
The Number Scientists Give You Instead of the Truth
A confidence interval is not telling you what you think it's telling you — and that misunderstanding has shaped medical decisions, policy debates, and scientific careers for decades.
The Idea
Here is the standard mistake. A researcher publishes a result with a 95% confidence interval, and almost everyone — including many scientists — reads it as: 'there's a 95% chance the true value is inside this range.' That feels intuitive. It's also wrong. What the interval actually says is stranger and colder: if you ran this exact study an infinite number of times and calculated the interval each time, 95% of those intervals would contain the true value. This particular interval either contains it or it doesn't. Probability, in the frequentist framework that produces confidence intervals, doesn't apply to single fixed facts — only to long-run processes. The confusion isn't just semantic. It matters enormously. The confidence interval is a statement about your procedure, not about this result. It tells you how well your method performs over many hypothetical repetitions — not how confident you should be in the specific number in front of you right now. This is why a narrow confidence interval can still be wildly misleading if the study design was flawed, the sample was unrepresentative, or the measurement was noisy. The interval faithfully reports the precision of your method while staying completely silent about its accuracy. Precision and accuracy are not the same thing, and a confidence interval only speaks to one of them.
In the World
In 2011, a team of social psychologists published a study claiming that people could predict the future — at statistically significant levels. Daryl Bem's paper in the Journal of Personality and Social Psychology reported that students performed better than chance at guessing which curtain would hide an erotic image, even before the computer had randomly selected it. The confidence intervals were narrow. The p-values were below the sacred 0.05 threshold. By every conventional metric, the results looked solid. What followed was one of the most instructive implosions in modern psychology. Replication attempts failed, almost universally. The problem wasn't that Bem had lied. The problem was that his methods — and the statistical tools he used correctly — were perfectly designed to produce impressive-looking intervals around effects that weren't real. He had run many variations of his experiments and reported the ones that worked. Each reported confidence interval was technically valid for that specific test. But the process that generated which test got reported was deeply biased. The interval measured the precision of a single chosen analysis while hiding the dozens of discarded ones behind it. This episode helped ignite the replication crisis across psychology and beyond, and it exposed with unusual clarity how confidence intervals can give a false sense of solidity to findings that are, in reality, built on sand.
Why It Matters
Most of us encounter confidence intervals not in academic papers but in news headlines: a new drug reduces risk 'by between 3% and 17%,' or economic growth is forecast 'at 2.1%, plus or minus 0.4 points.' The temptation is to treat the range as a region of roughly equal plausibility — a comfortable corridor of likelihood. But now you know that the interval is really a claim about a method's track record, not a map of where the truth probably lives. That reframe should make you ask different questions. Instead of 'is the true value inside that range?', ask: 'how many times was this type of study run, and how were the results selected?' and 'does this interval reflect precision or accuracy?' Those are harder questions, and they don't always have accessible answers. But just holding the distinction — knowing that statistical precision is not the same as being right — gives you a genuinely more sophisticated relationship with quantitative claims. In a world where data is constantly being handed to you as a substitute for argument, that's not a small thing.
A Question to Ponder
When you next encounter a statistical range presented as reassuring evidence, can you tell whether it's measuring how reliable the method is — or how likely the specific conclusion is to be true?
Get a new one of these every morning.
Start learning with Thinkable