Autonomous Vehicles
Why Self-Driving Cars Struggle with the Obvious
A toddler can tell a plastic bag from a tumbleweed rolling across a road — a state-of-the-art autonomous vehicle, given the wrong conditions, cannot.
The Idea
Autonomous vehicles perceive the world through a fusion of sensors — cameras, radar, and LiDAR (which bounces laser pulses off objects to build a 3D map of the surroundings). The hardware is, by now, genuinely impressive. The harder problem is what happens after perception: understanding what something *means* in context. Humans drive with a vast, mostly unconscious library of social and physical intuition. We read the body language of a pedestrian who is about to step off the kerb before their foot moves. We know that a ball rolling into the road probably has a child behind it. We negotiate four-way stops through eye contact and micro-gestures. None of this is formally taught — it is absorbed through years of being a body moving through the world among other bodies. For an autonomous system, every one of these judgements has to be either hardcoded as a rule or learned statistically from enormous datasets. Rules break on edge cases. Statistical learning works well in conditions resembling the training data and quietly fails on anything genuinely novel. This is sometimes called the 'long tail' problem: the vast majority of driving is routine, so routine scenarios get learned well, but rare, weird, high-stakes moments — a mattress on the motorway, a construction worker signalling manually, a child on a bicycle swerving unpredictably — are exactly the moments where the system has seen the fewest examples and faces the most danger.
In the World
In 2018, an Uber self-driving test vehicle struck and killed Elaine Herzberg as she walked her bicycle across a road in Tempe, Arizona — the first recorded pedestrian fatality involving an autonomous car. The car's sensors detected her nearly six seconds before impact. The software, however, classified her repeatedly and incorrectly: first as an unknown object, then as a vehicle, then as a bicycle. It couldn't settle on what it was seeing because the scenario — a person pushing a bicycle diagonally across a lane, at night, outside a crosswalk — appeared rarely enough in training data that the system had no confident classification to reach for. When there is no confident classification, the system defaults to inaction rather than emergency braking — a design choice intended to prevent phantom stops that might cause rear-end collisions. The logic, in isolation, seems reasonable. Applied to a real person crossing a real road, it was fatal. The incident revealed something important: the bottleneck in autonomous vehicles is not raw sensor capability but semantic understanding — the ability to interpret ambiguous real-world situations correctly and act on incomplete information under time pressure. It's essentially the same cognitive challenge that makes human driving hard to learn and hard to formalise: the world is genuinely ambiguous, and survival often depends on making the right call before you're certain.
Why It Matters
It's tempting to frame autonomous vehicles as a finished technology being slowly rolled out, with regulatory caution and public scepticism as the main brakes on progress. The more accurate picture is that the core technical problem — robust generalisation to novel, high-stakes situations — remains genuinely unsolved, and the industry has quietly shifted from 'full autonomy is two years away' to 'full autonomy is conditionally available in pre-mapped, geofenced areas.' That reframe matters beyond cars. It reflects something true about the current limits of AI more broadly: systems that perform brilliantly on well-represented scenarios can fail catastrophically on the unfamiliar, in ways that are hard to anticipate before deployment. The 'long tail' problem in autonomous driving is the same structural challenge facing AI in medicine, law, and finance. Knowing this, you can read coverage of autonomous vehicle milestones more critically — asking not just 'did the car complete the journey?' but 'what were the conditions, what was the fallback, and what happens when something genuinely unexpected occurs?' That question applies cleanly to a lot of AI announcements.
A Question to Ponder
If a technology performs flawlessly in 99% of situations but fails unpredictably in the remaining 1%, at what point — if ever — does that become acceptable in contexts where failure means someone dies?
Get a new one of these every morning.
Start learning with Thinkable