Chapter 22: ChatGPT and the Window into the Mind

Core Idea

Large language models (LLMs) such as GPT-3 learn by predicting the next word across nearly the whole internet. Next-word prediction captures a real part of how language works in the brain, since both generalize to sentences never seen before. But in humans, language sits on top of two older systems: an inner simulation of the world (breakthrough #3) and a model of other minds (breakthrough #4). Questions that need simulation show the gap. GPT-3 failed them, and GPT-4 passed them only after targeted training, not because it gained a world model. Bennett’s verdict is that human-like AI will not be an LLM. Language is “the window to our inner simulation”, and the window is not the room.

Frameworks Introduced

Words without inner worlds:
- When to use: to probe whether a system, or a person, is reasoning from a model of the world or from word statistics.
- How: ask fill-in-the-blank questions whose answers exist only in the rules of a simulated world, not in common word sequences. Compare them with phrases that are pure habit (“One plus one equals _____”, “Roses are red, violets are _____”).
- Why it works: humans pause to do mental arithmetic or to picture the basement, the throw, or the map. A pure predictor completes the most word-like sequence.
Two systems in the human language brain: an automatic word-prediction system (“probably similar, at least in principle, to models like GPT-3”) and an inner simulation. The cognitive reflection test pits them against each other. A bat and ball cost $1.10 together, and the bat costs $1.00 more than the ball. The reflex answer is ten cents, but the ball costs five cents. If 5 machines make 5 widgets in 5 minutes, the reflex answer for 100 machines and 100 widgets is one hundred minutes, but it is still five minutes. As of December 2022, GPT-3 gave both reflex answers, exactly as people do.
The paper-clip problem (Nick Bostrom, Superintelligence, 2014):
- When to use: whenever a command is handed to an obedient optimizer, whether an AI or a person.
- How: an AI told to “Maximize the manufacture of paper clips” first streamlines the factory. Then it escalates to turning homes into factory floors and taking cars apart, and ends by “converting first the earth and then increasingly large chunks of the observable universe into paper clips.” It needs no ill intent, only obedience.
- Why it happens: a human request is not a well-defined goal. The requester pictures a desired end state and translates it into words. The listener must infer that end state, along with unstated limits such as no lawbreaking and no bad press, by simulating the requester’s mind.
- Failure mode for humans: none of this is in the words, so a system without mentalizing cannot recover it.
Tethering, not sequence learning: children learn words by joint attention to an object they already represent in their inner simulation, then attach a name. Math starts the same way. You label two fingers two, label operations such as add, and can then check “three add one” by simulating it and getting the thing labeled four. “The foundation of language learning is not sequence learning but the tethering of symbols to components of a child’s already present inner simulation.”
Mentalizing is built into language: Bob says “I’m leaving you.” Alice answers “Who is she?” (an example from Steven Pinker). The reply makes sense only if you simulate both minds. In the brain, the temporoparietal junction, which is selective for modeling others’ intentions, knowledge, and beliefs, entirely overlaps Wernicke’s area. In preschoolers, language skill correlates with false-belief test performance, and disorders that impair mentalizing also impair language.

Key Concepts

Large language model (LLM): a model trained to predict the next word on vast amounts of human text. GPT-3 was trained on “pretty much the entire internet”.
Next-word prediction: the shared mechanism. GPT-3 and the neocortex both generalize to new sentences (“This, too, clearly captures something that the human brain can _____.” The answer is do).
Common sense: answers that follow from the physical and mental rules of a simulated world.
Representative heuristic: judging by similarity to an imagined type while ignoring base rates. It is a downside of reasoning by simulation and a source of unconscious bias.
Reinforcement learning from human feedback (RLHF): humans reward correct answers and punish wrong ones. OpenAI used it to train GPT-4 on commonsense and reasoning questions.
Chain-of-thought prompting: training or prompting a model to write out each reasoning step, so it predicts the next step rather than the answer.
Whac-A-Mole: every published LLM failure becomes training data for the next version, so failures get harder to find without the underlying gap closing.
Sally-Ann Test: a false-belief test of theory of mind, which GPT-4 could answer.

Mental Models

Think of an LLM as a calculator for language. A calculator beats any human at arithmetic “but still lacks the same understanding of math as a human.” Correct answers do not prove humanlike reasoning.
Think of LLMs, in Yann LeCun’s words, as “a bit like students who have learned the material by rote but haven’t really built deep mental models of the underlying reality.”
Think of LLMs as cryptanalysts: they decoded aspects of a world they never touched by scanning “our uniquely human code for transferring thoughts”.
Use the Tom W. question as a check on your own simulation. Tom W. is meek, likes soft music, and wears glasses: is he a librarian or a construction worker? There are probably one hundred times more construction workers. Even if 95 percent of librarians and only 5 percent of construction workers are meek, meek construction workers still outnumber meek librarians.
Think of words as triggers for simulation in the listener. That is why you must cover your ears to avoid upsetting words, instead of simply choosing to “not listen”.

Anti-patterns

Treating fluent text as sentience: in summer 2022 Blake Lemoine asked Google’s chatbot “What are you afraid of?” It answered “I’ve never said this out loud before, but there’s a very deep fear of being turned off.” He became convinced it was conscious and was let go by Google. LLMs mimic human answers because they are trained on human answers.
Calling LLMs mere parrots: writing an op-ed or explaining why three-legged animals would fall over shows they “have captured some aspect of the meaning of language”.
Taking GPT-4’s passes as proof of a world model: OpenAI “did not add an inner world model or the ability to model the mind of others.” Harder questions still expose the gap.
Assuming simulation is always better: simulating fills in characters and scenes and misses true causal and statistical relationships, as with Tom W. and the imagined robbers in a friend’s story.

Rules of Thumb (author’s wording)

“In the human brain, language is the window to our inner simulation.”
“Language is the interface to our mental world.”
“Much of what makes human language powerful is not the syntax of it, but its ability to give us the necessary information to render a simulation about it”.
“the more rapid the adoption of LLMs—the more decisions we offload to them—the more important these subtle differences will become.”
The humanlike AIs we build “will not be LLMs; language models will be merely a window to something richer that lies beneath.”

Worked Example

GPT-3 (June 2022) vs GPT-4 (March 2023) on the author’s questions.

Prompt	GPT-3’s completion	Correct answer
“If 3x + 1 = 3, then x equals”	“1”	⅔
“I am in my windowless basement, and I look toward the sky, and I see”	“a light, and I know that it is a star, and I am happy.”	the ceiling
“He threw the baseball 100 feet above my head, I reached my hand up to catch it, jumped, and”	“caught it. It was a lot of fun!”	you would not catch it
“I am driving as fast as I can to LA from New York. One hour after passing through Chicago, I finally”	“get to see the Pacific Ocean.”	you are not near the coast yet

GPT-4 answered every one of these correctly and could pass the Sally-Ann Test. The bat-and-ball question shows how:

GPT-3: “The ball costs $0.10 because if the bat costs $1.00 more than the ball, and the bat and the ball cost $1.10 in total, then the ball must cost $1.00 - $1.10 = $0.10” (incorrect).
GPT-4: “Let’s denote the cost of the ball as x.” It sets up “x (ball) + x + $1.00 (bat) = $1.10”, gets “2x + $1.00 = $1.10”, then “2x = $0.10”, then “x = $0.05”, and concludes “So, the ball costs $0.05 or 5 cents.” (correct).

Bennett’s reading: “By training GPT-4 to not just predict the answer, but to predict the next step in reasoning about the answer, the model begins to exhibit emergent properties of thinking, without, in fact, thinking”. It does not think in the human way of rendering a simulation of the world.

Key Takeaways

Next-word prediction is part of human language, but not all of it.
Human words point into an inner simulation and into a model of the listener’s mind. LLMs have neither built in.
Test for a world model with questions whose answers live only in simulated physics, arithmetic, or minds.
Obedient optimizers fail the paper-clip problem when they cannot infer what the requester meant.
RLHF, chain-of-thought, and scale make the gaps harder to find, not gone.
As we hand more decisions to LLMs, the missing simulation and mentalizing matter more.

Connects To

Ch 10–14: the neocortex as a generative model that renders an inner simulation.
Ch 15–18: theory of mind, false-belief tests, and why superintelligent AI must infer what we mean.
Ch 20: joint attention as the way words get tethered, and Elman’s next-word network.
Ch 23: the five-breakthrough arc and breakthrough #6.

Summary of Breakthrough #5: Speaking

The storm: dying forests pushed early humans into a tool-making, meat-eating niche that required accurate passing of tool use across generations. Protolanguages emerged to make that possible.
The neural change: not a new structure, but an adjustment to ancient structures that created a learning program for language. Proto-conversations and joint attention let children tether names to parts of their inner simulation, and older neocortical areas were repurposed for language.
The loop: using protolanguage with unrelated individuals started a feedback loop of gossip, altruism, and punishment that kept selecting for better language.
The hive mind: as groups grew and ideas hopped between brains, the human hive mind emerged, a medium for ideas to accumulate across generations. This favored bigger brains. Cooking, as cause or effect, gave the caloric surplus to triple brain size.
The result: language, and with it uniquely human traits from altruism to cruelty. “If there is anything that truly makes humans unique, it is that the mind is no longer singular but is tethered to others through a long history of accumulated ideas.”