Semiconductors & Hardware
The Bottleneck Baked Into Every Computer You've Ever Used
Every computer built in the last 75 years shares a fundamental design flaw that its own inventor recognised before the machine was even finished.
The Idea
In 1945, mathematician John von Neumann sketched out a design for how a computer should be organised: a single processor, a single pool of memory, and a single channel connecting them. Instructions and data would both live in that memory, fetched one at a time, processed, then sent back. It was elegant, general-purpose, and transformative — and it contained a structural problem that engineers have been fighting ever since. The problem is this: the processor is almost always faster than the memory. The CPU finishes its calculation and then waits — idle, spinning — for the next piece of data to travel across the bus from RAM. This gap, which grows wider with every generation of faster chips, is called the von Neumann bottleneck. It's less a bug than a consequence of the architecture's core assumption: that memory and processing are separate things, connected by a wire. The bottleneck doesn't just waste time. It wastes energy. Moving data is expensive — in power terms, often more expensive than the computation itself. As AI workloads have demanded processing of enormous datasets, this has become less an academic concern and more a fundamental constraint on what computers can do efficiently. The architecture that made modern computing possible is now, in certain demanding contexts, the thing holding it back.
In the World
The tension became impossible to ignore around 2015, when Google's engineers were trying to accelerate the neural networks behind services like Search and Translate. Their existing hardware — conventional CPUs, and even the GPUs borrowed from gaming — kept running into the same wall: the chips were fast, but feeding them data was slow. Vast amounts of time and energy were being spent just shuttling numbers back and forth across memory buses rather than actually computing anything. The response was the Tensor Processing Unit, or TPU — a custom chip designed from scratch to minimise the distance data travels. Instead of fetching operands from distant memory, the TPU keeps data flowing through a dense grid of small processors in a technique called systolic array processing: values ripple through the chip like a wave, each cell doing its calculation and immediately passing the result to its neighbour. Memory access drops dramatically. The bottleneck narrows. Google's first-generation TPU, deployed quietly in data centres in 2015 and only disclosed publicly in 2016, was reportedly 15 to 30 times faster than a contemporary GPU on inference tasks, while consuming a fraction of the power. It wasn't magic — it was a deliberate architectural choice to stop treating memory and computation as separate concerns. The von Neumann assumption had been loosened, and the machine got dramatically faster not because the transistors changed, but because the plumbing did.
Why It Matters
This is one of those ideas that quietly reframes how you think about technological progress. The dominant story of computing has been Moore's Law — transistors get smaller, chips get faster, everything improves. But the von Neumann bottleneck is a reminder that raw transistor speed is only part of the picture. Architecture matters. The assumptions built into a design at the beginning shape what's possible decades later. It also explains why the current wave of AI hardware — TPUs, neural processing units in phones, Apple's Neural Engine — isn't just marketing. These chips represent a genuine departure from the standard playbook, optimised for the specific pattern of computation that machine learning requires. When you hear that a new chip is purpose-built for AI, the von Neumann bottleneck is usually a large part of what's being designed around. More broadly, this is a useful lens for spotting hidden constraints anywhere: the elegant early solution that became a structural ceiling later. The question worth asking about any system — technical, organisational, social — is whether its original design assumptions still hold, or whether the world has moved fast enough to make them a liability.
A Question to Ponder
What other systems in your life — not just digital ones — are still running on architectural assumptions made at their founding, assumptions that nobody has seriously questioned since?
Get a new one of these every morning.
Start learning with Thinkable