Technology and Inequality
The Algorithm That Knows You're Poor
The digital tools designed to help you often work hardest against you the moment you can least afford it.
The Idea
There's a term circulating among researchers that deserves wider attention: 'digital redlining.' The original redlining — where banks and insurers drew literal red lines around low-income and minority neighbourhoods to exclude them from services — was outlawed decades ago. But its structural logic never disappeared. It got an upgrade. Algorithmic systems used in credit scoring, insurance pricing, hiring, and housing now routinely incorporate 'proxies' — data points that don't explicitly mention race, income, or postcode, but correlate with them so tightly that the effect is the same. The zip code you live in. The device you browse on. The time of day you apply for a loan. Whether you use prepaid mobile data instead of a monthly contract. Individually, these seem neutral. Aggregated, they reconstruct the very categories the law forbids. What makes this especially difficult to challenge is that it happens inside proprietary systems that no regulator, journalist, or affected person can fully inspect. Unlike the original redlined map — which could at least be photographed and published — these new boundaries are invisible, constantly recalibrated, and wrapped in the legitimising language of data science. The inequality they produce isn't an unfortunate side effect; it's often a feature. Riskier customers, in the system's cold arithmetic, are the ones who were already disadvantaged. The algorithm agrees with the prejudice, and calls it objectivity.
In the World
In 2019, researchers at the University of California, Berkeley, published an analysis of more than 31 million mortgage applications and found that both Latinx and Black borrowers were charged significantly more for home loans than white borrowers with identical financial profiles — not by human loan officers making conscious decisions, but by algorithmic systems. The gap persisted even after controlling for credit score, loan size, and income. The algorithm, it turned out, was using neighbourhood-level data that closely tracked racial composition. Around the same time, a team at MIT and Stanford demonstrated that commercial facial recognition systems — the kind sold to police departments and employers — misidentified darker-skinned women at error rates up to 34 percentage points higher than for lighter-skinned men. These weren't fringe products. They were market leaders. Joy Buolamwini, one of the researchers, had stumbled on the problem personally: the facial analysis software she was using for a project at MIT's Media Lab couldn't detect her own face until she held up a white mask. These aren't isolated failures. They are what happens when you train a system on historical data generated by a historically unequal society, and then deploy it at scale without asking what patterns, exactly, you're teaching it to reproduce. The system learns from the world as it was and helps make the world stay that way.
Why It Matters
Most conversations about AI and inequality stay at the level of principle — fairness is good, discrimination is bad. What's more useful is developing a specific instinct for where these dynamics actually land. When a service offers you 'personalised' pricing, it is deciding how much it thinks you'll pay — and that estimate is built from proxies that correlate with your economic position. When a hiring algorithm screens your CV, it may be pattern-matching against a historical workforce that was already filtered by bias. When a credit decision comes back as a clean, automated 'no,' there may be no human to appeal to and no explanation you're legally entitled to receive. This isn't an argument against algorithmic systems, which can also reduce certain kinds of human bias. It's an argument for holding them to the same standard we'd hold a human decision-maker — and for noticing that 'the algorithm decided' is increasingly used as a conversation-ending phrase when it should be a conversation-opening one. The next time a system tells you something about yourself, it's worth asking: whose world did it learn from?
A Question to Ponder
If an algorithmic decision about you turned out to be wrong, would you have any practical way to find out — and would you have any real power to challenge it?
Get a new one of these every morning.
Start learning with Thinkable