The Empty Room
Why the understanding we hand to machines is the kind we can no longer earn back
TL;DR: A policy on how to use AI is written by AI, summarised by AI, and commented on by AI, and no human in the chain understands a word of the thing they are now being governed by. Philosopher John Searle's Chinese Room Experiment imagined a room that mimics understanding for an audience that already has that understanding. Today, we have built the reverse. Increasingly, we have an audience that mimics understanding and hands the illusion of meaning to a machine that has no ability to understand at all. The point of this week's Thursday Thought is that the loss is not only that understanding itself disappears. It is the ability to earn understanding in the first place! Once a few generations stop knowing how to earn understanding, there will be very few people left who know what was ever there to be earned in the first place. Conquerors once burned a vanquished nation's books to ensure it forgot its history. We are now doing it to ourselves, and chalking it down as productivity.
“Nothing vast enters the life of mortals without a curse.” — Sophocles, Antigone
The Room, and the Room in Reverse
In 1980, the philosopher John Searle asked us to imagine a room. Inside that room sits a person who speaks no Chinese, but they hold a vast book of rules. As Chinese text characters are pushed in under the door, the person in the room looks them up in their book. They follow the rules, match one symbol to another, and create a perfect reply and push the response back out under the door. Those waiting outside quite reasonably accept that the person in the room understands Chinese. However, on the other side, there is no understanding at all, there is only a person moving symbols they cannot read. The room (unbeknownst to itself) has mimicked an understanding of Chinese for the people outside, who genuinely possess the understanding. Searle’s point was that this process is all a computer can ever do. It arranges symbols by the rules it has been programmed with. It never once understood what they mean.
The room was Searle’s answer to a test proposed thirty years earlier. Alan Turing had suggested that if a machine could hold a conversation no one could tell from a human’s, we should grant that it thinks, or else stop asking the question. The room is built to pass exactly that test and fail at another one. It converses perfectly but comprehends nothing. Passing for a mind, Searle argued, is not the same as having one.
Now let’s reverse the dynamic of the room.
The audience remain on the outside, but the flip is that they are now the ones who do not understand Chinese.
They push requests for output under the door and trust that the machine inside grasps it. Remember the room has always been empty! The machine they are outsourcing the reading to cannot read itself. The machine’s only understanding of reading is what it was once programmed with and the context provided by the sources it can access today.
In Searle’s original Chinese room, it is the one person inside the room who lacks understanding while the crowd outside possess a genuine understanding.
In the reverse (what I call the empty room), it is the crowd that has no understanding. They have gradually and unwittingly handed whatever understanding they had left to a machine that itself never had any understanding to begin with. An illusion of fluency travels both ways under the door, without any true understanding for either side.
We have already sat in this room, perhaps unbeknownst to ourselves. And if we have not, we will sit there very soon. The question is if we’ll even be aware of the room at all. Will anyone call it out? Or be aware to call it out at all?
Consider a committee who sits down to discuss a document they have collaborated on and “produced” together, let’s imagine it is a policy on the responsible use of AI.
The Signed-off Policy Nobody Actually Read
”Am I working for the model, or is the model working for me?” - Sangeet Paul Choudary, The Innovation Show EP 659
The policy lands in a dozen inboxes on a Tuesday morning, tick, another board action item off the list. One person was the lead in writing it, or rather, one person was the lead in prompting a model to write it. When done, they made it sound more like the company documents, skimmed it, and forwarded it on. The recipients feed the document into their own AI agents, which each return their own summary. Colleagues reply with comments (also AI-generated) and the comments are summarised in turn by the next reader (or more likely, their AI of choice) down the line.
By Friday close of play the policy offers an illusion that it has been circulated, discussed, approved, and adopted. It is then saved to the company shared drive, the cloud drive everyone can access but nobody really does. The policy is uploaded along with all the others behind a vault door the whole company has the key to but no one ever opens. It is more akin to a tomb than a shared drive, though not a tomb of documents, because the word document implies a reader who might one day stand over it and make the effort to absorb the words. It is closer to a tomb of data, of 1s and 0s, and the only thing that ever accesses the data is a machine, which reads that language and no other. It accesses the data not to understand the policy but to execute a prompt to summarise that data for the next person who will not take the time to read it either.
The regular visitor to the vault is the one reader who cannot understand a word.
Nobody, not human, nor machine in the loop, has read it. More precisely, nobody in the chain has understood the thing they are now being governed by. Yes, a document exists, yes it was co-created, but no, the comprehension it was produced to maintain doesn’t exist at all.
This is what I call the silhouette of work, an outline of understanding cast on a wall while the understanding itself was never earned. Everyone appears to be busy and up to speed (and they may even believe they are). Meetings happen, sign-offs land, policies are neatly stored in the folder where policies go. And somewhere in the middle of all that motion sits a model that drafted the rules for its own use, arranging clauses about care with no more grasp of what care means than it has of a Thursday, or of time passing at all. It has a concept of neither, but casts the illusion of both.
The policy is all but signed off.
Next stop, an audit and risk committee, then the board, the people whose job is to have read it, whose signatures and approvals suggest they have. But here’s the thing, they have also done the same thing everyone below them has.
They fed it to a model, took the summary at face value, and approved. The last line of defence turns out to run with the same plays as the first. As the policy travels up the chain, fewer humans touch it, until it is ratified at the top by nobody at all.
So the apparent meaning gets passed like a baton from hand to hand, but really no hand ever comes close to a true comprehension at all.
Afterwards, the organisation may even believe they wrote it. Deep down, some will know the truth, but they will shush the discomfort the way we all do, by deciding that prompting the machine was the work and the judgement was theirs, the typing was but a task to be outsourced. This is the self-justification Carol Tavris and Elliot Aronson traced through every field where people need to keep seeing themselves as capable and honest, in Mistakes Were Made (But Not by Me).
We always rearrange the story until the discomfort is balanced. Most others will not justify anything. They will truly believe they created the policy, because the part of them that once knew the difference between making a thing and requesting it has atrophied so much that those now feel the same. They used today’s tools cleverly, or so they will justify to themselves. And they may well be telling the truth, as far as they understand it today.
None of this is new. Knowledge has decayed for generations.
The Island That Forgot Fish
“Institutional memory, and its attendant facts and knowledge, are only as permanent as its generation time.”— Samuel Arbesman, The Half-Life of Facts
At the end of the Ice Age, as the planet warmed, glaciers melted and sea levels rose. Bit by bit, the ocean swallowed the land bridge that once connected Tasmania to the Australian mainland. What had been a walkable crossing across what we now call the Bass Strait turned to open water. And like the land itself, the flow of ideas, tools, and stories slipped beneath the waves.
Once Tasmania stood isolated, its toolkit began to shrink. Bone needles, fishing gear, even the act of fishing itself, all faded from memory. By the time European explorers arrived in the seventeenth century, they found the Tasmanians using just two dozen tools, mostly rocks, clubs, and other basics. Meanwhile, Aboriginal communities on the mainland retained hundreds of technologies: boats, barbed spears, fishing nets, cold-weather clothing.
The fish had not left the water. What had left was the know-how of catching fish, and the practice atrophied because it stopped being carried forward. A skill only truly lives in the people who keep doing it. Once a few generations had grown up without earning how to do it, there was no one left alive who earned the knowing, and bit by bit the skill dwindled amongst the population one death at a time until it reached zero. The anthropologist Joseph Henrich has shown how this works in small, isolated groups. Skills go unpractised, new ideas do not spread, and eventually even once-common knowledge simply vanishes.
Nobody woke one morning and decided to forget fishing. Such loss is invisible while it is happening, because at every step the community still eats, still survives, still appears competent at the things it keeps alive.
There is a smaller island off the coast of Tasmania called Flinders Island, cut off in the same way at the same time, and there the process ran all the way to its end.
About four thousand years after its land bridge was destroyed, the population did not merely lose its tools. It vanished entirely, in what the economist Michael Kremer called a technological regress, the loss of even the technologies basic for survival.
Nothing Left to Pour
“From shirtsleeves to shirtsleeves in three generations.” — proverb
The first generation makes the money, the second spends it, and the third is left with none. The saying turns up in almost every culture, which is usually a sign that people have watched the same pattern unravel too often to call it coincidence. The Lancashire mills said clogs to clogs in three generations, and the Chinese said that wealth never survives three generations. They all noticed it, but nobody found a reliable way to stop the rot.
The easy explanation is that the grandchildren were careless with the money. The harder to accept one is that the money was not the true inheritance. What the first generation built was the capacity to build, forged in the act of building, and that capacity is the one thing that cannot be handed down, because it can only live in the muscle memory of the person who did the work.
The second generation receives the results without the earning, and the third receives neither, so far removed from the struggle to ever understand it. In the end, the family does not lose the money and then the skill. It loses the skill first, and the money follows that loss. This is the same way the island lost the practice and then the fish and the same way we lose the ability to understand.
In a previous Innovation Show, the physicist David Deutsch told us that education is not simply a pouring of knowledge from one generation to the next. We may picture education, and inheritance, and handover, as pouring. Let’s just imagine for a moment that knowledge is a liquid. One generation fills a vessel and pours it into the next and teaching is the act of tipping knowledge across. When you internalise it that way, you might think of the liquid as the thing that matters most and perhaps the pouring is just a means.
The island teaches us otherwise.
Knowledge doesn’t travel because it is poured. It travels because each generation has to re-earn it in the actual doing, hands in the water, learning the tide, cutting the hands while gutting the fish over and over again until it becomes tacit. The vessel is refilled every time by the person earning it. The act, sometimes struggle, of refilling is where the transmission lives. There is no shortcut where the full vessel simply tops up the empty one. The effort is not the cost of the knowledge. The effort IS the knowledge imprinting.
This is why the AI policy tale is a different kind of loss than it might first appear.
On the surface a task gets done faster, 1-0 to productivity and the company appears more efficient! Below the surface, the pouring itself has been automated, and there is never any liquid in the pour.
The vessel on the receiving end stays empty while it appears to be full.
Learning to Earn
The learning pattern grows the way a tendon grows, not the way a muscle grows. Give a body steroids and the muscle grows in weeks while the tendon that anchors it thickens over months. Thus, the strength arrives ahead of the structure able to carry it, and the tissue tears under its own new power.
Understanding is like tendon growth, not the muscle building. You learn only through earning it. Likewise, with knowledge work, if you skip the struggle work then you never build the apparatus that reaches facts, tests them, and knows when one has gone wrong.
The trap here is that answers will keep arriving, fluent and ahead of time. The real danger is that the population able to question them keeps shrinking, one un-earned answer at a time, in exactly the invisible way the island lost fish. Everyone looks productive. The policy gets approved. The room appears full, but is empty at the same time.
What Grew There Once Before
“We are missing insight into the process by which it came to be an answer.” — Samuel Arbesman, Overcomplicated
The last person on the island who knew how to fish did not know they were the last. Like the last time a child will hold our hand, or the last time we see a parent before they die. These last times never announce themselves. They are only ever visible from later, from the far side of the loss, when the hand is grown too old and the chair is truly empty. That is the sinister pattern of this kind of forgetting. Like coastal erosion, it makes no sound, because at every step the work still gets done and the sea still looks the same. Baseline after shifting baseline, our understanding dwindles.
Conquerors have always understood this.
When they wanted a nation gone, they did not stop at tearing down walls. They destroyed its memory. The Spanish burned the Maya codices at Maní, and a civilisation’s written records, centuries of astronomy, history, and belief, went up in one afternoon’s fire. The Mongols threw the House of Wisdom into the Tigris, and it was said the river ran black with the ink of its books. Rome salted the ground where Carthage once thought. Every conqueror executed the same strategy. If you just kill the soldiers, a nation resists; but if you burn the library, the nation forgets it was ever a nation at all. It was an act of violence. Everyone could see it was. The people wept as the books burned, because they knew exactly what was being taken from them.
We will not weep.
That is the difference. No invading army is marching towards our understanding. We are letting it slide out of the library ourselves, one un-earned answer at a time, and stacking it neatly in a vault we no longer open.
The fires this time are efficient, silent, and self-lit. And when the earning of learning has finally been handed over in full, the minutes will record that productivity had never been higher.
The answers will continue come, increasingly fluent and ahead of schedule. The real question is whether anyone will be left who remembers there was something to earn.
On This Week’s Innovation Show Episode 659
Two guests, one question that runs straight through this Thursday Thought. Rita McGrath returns alongside Sangeet Paul Choudary to explore what AI does to strategy itself, not the tools of the game, but the game.
Rita traces how company value has flipped from tangible to intangible inside a working lifetime, and why the old strategy frameworks assumed a fixed board that no longer exists.
Sangeet asks the question that became this essay’s epigraph. Am I working for the model, or is the model working for me? His warning about fluency, that polished output can make us feel we understand things we could not take apart, is the empty room described from the inside.
Listen here:
YouTube coming soon.
