Statistics & Data
The Bullet Holes That Weren't There
In World War II, the planes that came back riddled with bullet holes were the ones telling the Air Force exactly where not to add armour.
The Idea
Survivor bias is the statistical error of drawing conclusions from only the things that made it through a filter — while remaining blind to everything the filter removed. It sounds obvious once named, but it runs so deep in how we process information that even trained scientists routinely fall for it. The logic is seductive: you look at what's in front of you, notice patterns, and draw conclusions. The problem is that what's in front of you is never a random sample. It's the survivors. Everything else — the failed experiments, the bankrupt companies, the lost ships, the dead patients — is invisible by definition, and its absence warps every conclusion you draw from the data you can see. This matters because the bias doesn't just introduce random noise; it systematically points you in the wrong direction. When you study only successful outcomes, you reliably overestimate the factors that correlate with success. When you study only existing entities, you miss the crucial question of what's no longer there and why. The ghost data — the missing cases — carries as much information as the visible data, sometimes more. Good statistical reasoning means developing an almost uncomfortable habit: constantly asking what you're not seeing, and why it isn't in your sample.
In the World
The classic case comes from Abraham Wald, a Hungarian-born mathematician working with the Statistical Research Group at Columbia University during World War II. The US military had a problem: aircraft were being shot down over Europe, and they needed to know where to add extra armour. Extra armour meant extra weight, which meant worse performance, so they couldn't reinforce everything. They needed to be precise. Engineers examined the bombers that returned from missions and mapped where the bullet holes clustered — mostly in the fuselage and wings, far fewer in the engines and cockpit. The intuitive conclusion: reinforce the fuselage and wings. That's where the planes were getting hit. Wald saw it differently. The planes in front of them had survived. The bullet holes they were measuring showed where a plane could take damage and still make it home. The areas with almost no bullet holes — the engines, the cockpit — weren't being missed by enemy fire; they were the places where a hit meant the plane never came back at all. The missing planes were the data. Wald recommended reinforcing the areas that looked unscathed. His insight was only rediscovered and widely publicised decades later, largely through the writer Jordan Ellenberg's retelling. For most of the intervening years, it sat in a classified wartime report — a perfect survivor of a story about survivors.
Why It Matters
Once you internalise this error, you start noticing it in places that genuinely change how you evaluate advice and evidence. Every entrepreneur who tells you persistence and passion are the keys to success is, by definition, someone who persisted and is passionate — and made it. The thousands who were equally persistent and passionate and still failed are not on stages or podcasts. Every book about what elite universities have in common is written about universities that became elite and remained so. The ones that tried the same things and didn't make it aren't in the dataset. This doesn't mean success stories have nothing to teach you — it means they can only teach you what's necessary, not what's sufficient. Bullet holes in the fuselage are necessary to study; they're just not where the actionable insight lives. The habit survivor bias trains you toward is a kind of productive suspicion: when a pattern looks clean and consistent, ask who or what had to disappear for it to look that way. It's less about cynicism than about accounting for the full picture — including the part the world has quietly edited out.
A Question to Ponder
What's one belief you hold about success, health, or human nature that might look very different if you could somehow see all the cases that didn't make it into your awareness?
Get a new one of these every morning.
Start learning with Thinkable