Algorithmic Bias
The Mirror That Lies: Why AI Reflects the Past Instead of the Future
An algorithm used by US courts to predict criminal reoffending was twice as likely to falsely flag Black defendants as high-risk compared to white defendants — and it was considered state of the art.
The Idea
Algorithmic bias is easy to misunderstand. The common assumption is that bias sneaks in because someone coded their prejudices directly into the system — a bad actor writing bad rules. The more unsettling truth is almost the opposite: bias emerges most reliably from the data itself, even when every engineer involved has the best intentions. Machine learning systems learn by identifying patterns in historical data. The problem is that history is not neutral. It is a record of decisions made under conditions of inequality — who got hired, who got loans, who got bail, who got diagnosed. When a model trains on that record, it doesn't just learn patterns; it learns to reproduce the logic that generated those patterns, including the discriminatory logic baked into every past decision. This creates a feedback loop that is genuinely hard to escape. A hiring algorithm trained on a decade of successful employees will learn to prefer candidates who resemble past hires — who, in many industries, skew heavily toward particular demographics. The model isn't being racist in any intentional sense. It is being precisely what it was designed to be: accurate to the data. The data just happens to encode a world we would rather not replicate. What makes this especially thorny is that removing protected characteristics — race, gender, age — from the dataset often isn't enough. Zip code, word choice in a CV, the school someone attended: these can all act as proxies, carrying the discriminatory signal through a side door.
In the World
In 2016, ProPublica published an investigation into a risk-assessment tool called COMPAS, which was being used by judges in several US states to help determine bail and sentencing decisions. The tool scored defendants on their likelihood of reoffending, and those scores were influencing whether real people went home or went to jail. ProPublica's analysis of more than seven thousand defendants in Broward County, Florida, found something striking. COMPAS was not significantly more accurate at predicting reoffending for white defendants than for Black defendants overall — but its errors were distributed in a deeply unequal way. Black defendants who did not go on to reoffend were nearly twice as likely to be incorrectly flagged as high-risk. White defendants who did reoffend were more likely to have been incorrectly labelled low-risk. The company behind COMPAS, Northpointe, disputed the analysis, and a separate group of researchers argued that ProPublica was using the wrong statistical definition of fairness — a debate that revealed something genuinely uncomfortable: there is no single mathematical definition of fairness that satisfies all conditions simultaneously. You can tune a model to equalise false positive rates across groups, or to equalise accuracy, but doing both at once is, under most real-world conditions, mathematically impossible. Judges, meanwhile, were treating these scores as if they were objective measurements rather than probabilistic outputs trained on historical inequity. The algorithm's veneer of scientific precision was, in practice, laundering a very old kind of judgment.
Why It Matters
It would be convenient if algorithmic bias were a purely technical problem with a technical fix — better data, cleaner pipelines, fairer training sets. Some of it is. But the deeper issue is about what we ask algorithms to do in the first place. When an algorithm is tasked with predicting future behaviour from past patterns, it is always, in some sense, being asked to argue that the future should resemble the past. In domains where the past was shaped by structural disadvantage, that is not a neutral request. This matters for how you read almost any headline about an AI system making consequential decisions — in hiring, lending, healthcare, policing, or content moderation. The question to ask is not just 'is the model accurate?' but 'accurate for whom, and at whose expense?' Aggregate accuracy can mask radically different error rates across groups, and it is usually the already-disadvantaged group that absorbs the false positives. Being aware of this doesn't make you cynical about AI — it makes you a more precise reader of it. The technology is genuinely powerful. That's exactly why the assumptions embedded in its design deserve scrutiny rather than deference.
A Question to Ponder
If a system is statistically accurate on average but consistently wrong about specific groups, at what point does calling it 'fair' become a way of avoiding a harder conversation?
Get a new one of these every morning.
Start learning with Thinkable