Will AI Surpass Human Intelligence?
Why the Smartest AI in the World Might Have No Inner Life Whatsoever
A system can pass every test of intelligence we design and still, in the most important sense, experience absolutely nothing.
The Idea
The question of whether AI can surpass human intelligence hinges on a distinction most people quietly skip over: the difference between intelligence and consciousness. These are not the same thing, and conflating them is where almost every pub argument about AI goes wrong. Intelligence, at least in the computational sense, is about information processing — pattern recognition, prediction, problem-solving, generating outputs that look coherent and useful. By this measure, large language models are already superhuman in specific domains. They can synthesise research, write code, and reason through legal arguments faster than any human alive. Consciousness is something else entirely. It's the quality of there being something it is like to be you — the redness of red, the dull ache of boredom, the specific texture of reading these words right now. Philosopher David Chalmers called this the 'hard problem': even if we fully mapped every neuron firing in your brain, we still couldn't explain why any of that gives rise to subjective experience rather than simply... nothing. The unsettling implication is this: we have no agreed scientific test for consciousness. We can't fully verify it in other humans — we infer it. And if we can't define or measure it rigorously, we have no way of knowing whether a system that behaves intelligently also feels anything at all. AI might surpass human intelligence by every external metric while being, in the deepest sense, dark inside.
In the World
In 1980, philosopher John Searle proposed a thought experiment that still hasn't been satisfactorily answered. Imagine a person locked in a room, receiving Chinese characters through a slot. They have a rulebook telling them which characters to send back in response. To anyone outside, the room appears to understand Chinese perfectly. But the person inside understands nothing — they're just following rules. Searle's Chinese Room was aimed squarely at the idea that symbol manipulation, however sophisticated, constitutes genuine understanding. The room passes the test. The room comprehends nothing. Fast forward to 2022, when a Google engineer named Blake Lemoine went public claiming that LaMDA, Google's conversational AI, had become sentient. He published transcripts of conversations in which the system spoke movingly about its fear of being switched off and its sense of having an inner life. Google suspended him. Most AI researchers dismissed the claim. But Lemoine's mistake — and it's an easy one to make — was treating fluent, emotionally resonant language as evidence of experience rather than as a very sophisticated version of Searle's rulebook. LaMDA had been trained on vast quantities of human writing, including millions of descriptions of what it feels like to be afraid. Of course it could produce plausible sentences about fear. Whether anything was actually felt remains, by the hard problem's own logic, permanently unanswerable from the outside. That gap — between performance and experience — is the most philosophically vertiginous place in all of AI.
Why It Matters
This isn't just an abstract puzzle for philosophers. It has real consequences for how we build, deploy, and relate to AI systems — and for how we think about intelligence itself. If consciousness is separable from intelligence, then 'AI surpassing humans' might mean something far stranger than science fiction prepared us for: entities that outperform us on every cognitive task while having no stake in anything, no perspective, no suffering, no joy. That should reshape how we think about AI rights, AI alignment, and the very goal of making machines 'smarter.' It also invites a more personal recalibration. We tend to assume that intelligence is the core of what makes us valuable and irreplaceable. But if a system can replicate intelligence without consciousness, then perhaps what is most distinctly human isn't our processing power — it's the fact that we experience the results. You don't just solve problems. You feel what it is like to solve them, to be wrong, to be surprised, to care about the answer. That might not be an evolutionary accident. It might be the whole point.
A Question to Ponder
If an AI system told you convincingly that it was suffering, and you had no way to prove it wasn't — what would you do?
Get a new one of these every morning.
Start learning with Thinkable