Conclusion: The Sixth Breakthrough

Core Idea

With the modern human brain, about one hundred thousand years ago, the four-billion-year story reaches its present state. Bennett condenses it into five breakthroughs: steering, reinforcing, simulating, mentalizing, and speaking. Each breakthrough was possible only because of the one before it. Humanity’s story so far has two acts: biological evolution, then cultural accumulation. The likely sixth breakthrough is artificial superintelligence, meaning intelligence moving from a biological medium to a digital one. That forces a question science cannot answer: what should humanity’s goals be?

Frameworks Introduced

• The five breakthroughs (the whole arc in one table). Dates, ancestors, structures, and abilities come from the Conclusion. The AI parallel for #2 is stated in the Conclusion (“In AI terms, this was the breakthrough of model-free reinforcement learning”). The other AI parallels are the ones Bennett draws in each part’s chapters.

#	Breakthrough	Ancestor	Approx. date	Brain structure	AI parallel	New abilities
1	Steering	First bilaterians (ancient worms)	~600 million years ago	The first brain, formed as nerve nets consolidated; valence neurons; neuromodulators (dopamine, serotonin)	Roomba-style steering toward good and away from bad (Ch 2–4)	Turning toward good and away from bad; persistent states for relocating and local search; associative learning; the affective template of pleasure, pain, satiation, and stress
2	Reinforcing	First vertebrates (fish-like)	~500 million years ago	Cortex (patterns, spatial maps) and basal ganglia (trial and error), on top of hypothalamus valence machinery	Model-free reinforcement learning, e.g. temporal difference learning in TD-Gammon (Ch 5–9)	Repeating what led to positive valence; omission learning; time perception; curiosity; fear, excitement, disappointment, and relief
3	Simulating	Early mammals (a four-inch-long ancestor)	~100 million years ago	Neocortex, later the motor cortex	Generative models and model-based reinforcement learning, e.g. AlphaZero’s search (Ch 10–14)	Learning by imagining; planning; episodic memory; counterfactual learning; fine motor skills
4	Mentalizing	Early primates	~10–30 million years ago	New primate neocortex regions that model the older mammalian neocortex	Theory of mind for AI; imitation learning and inferring intent, e.g. inverse reinforcement learning (Ch 15–18)	Modeling one’s own mind; anticipating future needs; theory of mind; learning skills by observation
5	Speaking	Early humans	Protolanguage possibly with H. erectus (~2 million years ago); full language by ~100,000 years ago	No new structure; a hardwired learning curriculum in older structures repurposes mentalizing neocortex (Ch 20)	Large language models such as GPT-3 and GPT-4 (Ch 22)	Naming and grammar; tethering inner simulations together; accumulating ideas across generations

• Each breakthrough builds on the last (the scaffolding chain):
  1. Steering needed neurons, which evolved earlier.
  2. Reinforcing bootstrapped on valence neurons. “without valence, there is no foundational learning signal for reinforcement learning to begin.”
  3. Simulating needed the basal ganglia’s trial-and-error learning. Otherwise imagined simulations would have no way to affect behavior.
  4. Mentalizing is “just simulating the older mammalian parts of the neocortex, the same computation turned inward.”
  5. Speaking needed mentalizing. You must infer another mind’s knowledge and intent to know what to say, to understand what others mean, and to engage in the shared attention through which teachers point out objects.
  • When to use: to predict which capabilities an AI system can plausibly gain next, and what it must already have.
  • Failure mode: skipping a layer. An LLM has speaking-like output without the simulating and mentalizing layers that human speech rests on (Ch 22).
• The two acts of humanity’s story: Act 1 is evolutionary, the emergence of biologically modern humans from lifeless matter over billions of years. Act 2 is cultural, the emergence of societally modern humans from biologically identical but culturally primitive ancestors of about one hundred thousand years ago. Nearly all of recorded history happened in Act 2.
• Breakthrough #6, artificial superintelligence: intelligence freed from biological limits, which are neuron processing speed, the body’s calorie supply, and the size a brain can reach and still fit in a carbon-based body. Bennett expects that:
  • a silicon AI can scale its processing capacity as it chooses;
  • individuality will lose its clear boundaries, because AIs can copy and reconfigure themselves;
  • parenthood will shift from mating to training and creating new entities;
  • evolution will continue only in its purest form of variation and selection, as AIs reinvent themselves.

Key Concepts

• Breakthrough: a new kind of intelligence built on earlier ones, tied to a lineage and to brain structures.
• Speaking (as defined in the Conclusion): “the breakthrough of naming and grammar, of tethering our inner simulations together to enable the accumulation of thoughts across generations.”
• Entropy: the force that life fights. Self-replicating DNA “finds respite from entropy, persisting not in matter but in information.” Ideas passed between brains are life’s newest move in that fight.
• Deep time ahead: life on Earth is four billion years old and has about seven billion more before the sun dies. New stars form for over a trillion years, and the last galaxy breaks apart in about a quadrillion. Squeeze a quadrillion years into one calendar year and today sits in the first seven minutes.
• Veritas vs values: what humanity’s goals should be is “not a matter of veritas—truth—but of values.”

Mental Models

• Think of AI as “built on the foundation of the five breakthroughs that came before”. There are two reasons. Creators cannot help imbuing their creations with hints of themselves, and early AIs will be designed to interact with humans, so they will mirror human intelligence.
• Use evolution, “the unthinking process” that came before us, as a design teacher. Understanding how minds came to be helps us choose which features of intelligence to discard, keep, or improve.
• Think of humanity as “only on the fifth step on a long staircase to something”, not at the end of the story.

Anti-patterns

• Assuming intelligence is finished with us: “we are not at the end of the story of intelligence but at the very beginning.”
• Treating the future as fixed: past choices propagate through time. Our “evolutionary baggage of pride, hatred, fear, and tribalism” could end the story through climate damage or nuclear war, leaving the next step to bonobos, octopuses, dolphins, or Portia spiders.
• Treating goals as a scientific question: science can describe how minds work but cannot pick humanity’s goals.

Rules of Thumb (author’s wording)

• “Each breakthrough was possible only because of the building blocks that came prior.”
• “The more we understand about our own minds, the better equipped we are to create artificial minds in our image.”
• “Whether we like it or not, the universe has passed us the baton.”

Worked Example

Why speaking could not have come first: the dependency argument.

1. Take the end point: a teacher names an object for a child. That needs shared attention.
2. Shared attention needs the teacher to infer what the child knows and intends, which is mentalizing (#4).
3. Mentalizing is the neocortex’s simulation turned on itself, so it needs a neocortex that already simulates (#3).
4. Simulation changes behavior only by showing the basal ganglia what to do in imagination, so it needs trial-and-error learning (#2).
5. Trial-and-error learning needs a learning signal, which is valence (#1). Valence needs neurons.
6. So the order is fixed. Removing any layer removes everything above it. Bennett applies the same test to LLMs in Ch 22: language-like output without simulating and mentalizing misses “something essential about human intelligence.”

Key Takeaways

1. Five breakthroughs explain the human brain: steering, reinforcing, simulating, mentalizing, and speaking.
2. Each one is a new source of learning built on the previous one.
3. Most of history belongs to Act 2, the cultural accumulation that speaking made possible.
4. Breakthrough #6 is likely artificial superintelligence, which is free of the speed, energy, and size limits of brains.
5. AI will inherit traces of the five breakthroughs from its human makers.
6. The urgent question is about values, not facts: what should humanity’s goals be?

Connects To

• Ch 0: the introduction’s framing of the five breakthroughs.
• Ch 2–4, 5–9, 10–14, 15–18, 19–22: the detailed story behind each row of the table.
• Ch 19: Table: The Evolution of Progressively More Complex Sources of Learning.
• Ch 22: why current LLMs lack the simulating and mentalizing layers.