Algorithmic Accountability
Who Do You Sue When the Algorithm Gets It Wrong?
When an algorithm denied Robert McDaniel a loan, flagged him as a future criminal, and got him visited by police — all before he had done anything — there was no one to call.
The Idea
Algorithms make decisions about us constantly — who gets credit, who gets an interview, whose social media post gets amplified, whose parole request gets approved. The people building these systems typically insist they are just surfacing patterns in data. And yet the patterns in data are the residue of human choices, often discriminatory ones, now laundered through mathematics and given a veneer of objectivity. This is the core tension at the heart of algorithmic accountability: when a decision causes harm, who is actually responsible? The software engineer who wrote the code? The company that deployed it? The client institution that acted on its output? Or the training data itself, reflecting a world we have not yet fixed? What makes this question especially difficult is that algorithmic systems are often deliberately opaque — not just technically complex, but legally and commercially shielded. In the European Union, there is now a right to explanation for automated decisions. In practice, this right is hard to exercise. Companies can cite trade secrets. The explanation a model can generate about its own output is not the same as a causal account of why that output occurred. Accountability requires a legible chain from decision to harm to responsible party. Algorithms are extraordinarily good at breaking that chain.
In the World
In 2016, ProPublica published an investigation into a recidivism scoring tool called COMPAS, widely used by American courts to help judges decide whether to grant bail or parole. The algorithm assigned defendants a risk score — effectively a prediction of whether they would reoffend. ProPublica's analysis found that Black defendants were nearly twice as likely as white defendants to be falsely flagged as high risk, while white defendants were more likely to be incorrectly labelled low risk. The company behind COMPAS, Northpointe, disputed the methodology. Academics split into factions arguing over which mathematical definition of fairness the tool should have been optimised for. What almost no one disputed was this: real people were receiving longer sentences or being held on remand based on a score they could not see, could not challenge, and could not understand. The judge in the room may have believed they were exercising discretion. In practice, many were anchoring on a number generated by a model trained on historical arrest data — data that, given the well-documented racial disparities in American policing, was anything but neutral. COMPAS is still in use. The debate it triggered, about who is accountable when an inscrutable tool shapes a human life, has not been resolved.
Why It Matters
Most of us will never end up in front of a sentencing algorithm, but we interact with consequential automated systems constantly — and usually without knowing it. A hiring platform screens out your application. A credit model quietly adjusts your rate. A content feed decides which version of the world you see most often. The accountability gap in each of these cases is the same: there is no clear mechanism for you to contest the decision, understand how it was made, or identify who bears responsibility for its effects. What shifts when you understand this is not helplessness — it is a more precise sense of where to apply pressure. Calls for algorithmic transparency are not abstract tech-policy concerns; they are arguments about who gets to challenge a decision that affects your life. And the frameworks being built now — audit requirements, impact assessments, rights to explanation — will either have teeth or they will not. Whether they do depends in part on whether enough people understand what is actually at stake.
A Question to Ponder
If a decision about your life was made by an algorithm, and that decision turned out to be wrong, what would a fair and realistic form of redress actually look like?
Get a new one of these every morning.
Start learning with Thinkable