Statistics & Data
Why Most Published Research Findings Are Wrong — And Why That's a Math Problem
A study with 20 participants and a thrilling result is almost certainly lying to you — not because the researcher cheated, but because the numbers were always going to do this.
The Idea
Statistical power is the probability that your study will detect a real effect if one actually exists. Think of it as the sensitivity of your scientific instrument. A study with low power is like trying to spot a faint star through a fogged-up telescope — even if the star is there, you probably won't see it. And crucially, if you do see something, you can't be sure it's real and not just a smudge on the lens. Here's what makes this genuinely unsettling: most studies in psychology, medicine, and social science have historically been underpowered — often dramatically so. A landmark 2005 paper by epidemiologist John Ioannidis argued that when studies are small and the effect being hunted is subtle, the majority of 'statistically significant' findings will be false positives. The maths are counterintuitive but hold up: if you run an underpowered study on a hypothesis that is only moderately plausible, a positive result is more likely to be a fluke than a discovery. Sample size is the lever that controls power. Larger samples reduce the noise in your data, making real signals easier to distinguish. But recruiting participants costs time and resources, so researchers routinely run studies that are just large enough to feel credible — which is not the same as being large enough to be trustworthy. The result is a scientific literature peppered with exciting findings that quietly fail to replicate when someone bothers to check.
In the World
In 2011, social psychologist Daryl Bem published a paper in a prestigious peer-reviewed journal claiming that people could perceive future events before they happened — precognition, in other words. The studies used standard methodology, passed peer review, and achieved statistical significance. The sample sizes were around 100 participants per experiment, which sounds reasonable. The problem was power. When other researchers attempted to replicate Bem's findings using larger samples — sometimes three or four times the size — the effects evaporated. A replication effort published in 2012 by Jeff Rouder and Richard Morey showed that the original studies were simply too small to reliably distinguish a genuine effect from random variation. Bem's results were almost certainly noise that the study's design had no way of filtering out. This wasn't an isolated scandal — it became one of the opening shots of what researchers now call the replication crisis. In 2015, the Reproducibility Project attempted to replicate 100 published psychology studies. Fewer than half held up. A follow-up analysis found that the original studies had, on average, only about 50% power — meaning that even if every hypothesis had been true, half the studies would have missed the effect anyway. The ones that did find something were disproportionately the flukes. The lesson wasn't that scientists are fraudulent. It was that the field had built its publication incentives around the wrong thing: finding results, not finding truth.
Why It Matters
You encounter the downstream consequences of underpowered research more often than you might realise. The news story about a food that prevents cancer, the management technique that boosts productivity by 30%, the personality trait linked to life expectancy — many of these originate in studies that were never equipped to deliver reliable answers. Knowing this doesn't mean dismissing science; it means reading it more carefully. A few useful instincts to carry forward: treat single studies as interesting leads, not conclusions. Ask whether a finding has been replicated, especially in a larger sample. Be more impressed by a null result from a well-powered study than by a dramatic positive result from a small one — the former takes genuine rigour. And when you read that a result is 'statistically significant,' remember that this phrase says nothing about whether the study was powerful enough to mean anything. The deeper shift is recognising that statistics is not just a tool for confirming hunches — it is a discipline that demands honesty about uncertainty. Power analysis, done properly, forces a researcher to commit to how large an effect they expect before they run the study. That pre-commitment is one of the most powerful antidotes to self-deception in all of science.
A Question to Ponder
If the incentives in a field reward publishing positive results over rigorous ones, what would it actually take to change the culture — and who would have to pay the price first?
Get a new one of these every morning.
Start learning with Thinkable