Chapter 12: Mice in the Imaginarium

Core Idea

The older vertebrate cortex could already recognize objects, so the neocortex’s original job was probably not recognition. It was imagining the world as it is not. Bennett suggests recognition was the means and simulation the payoff. Simulation gave early mammals three new abilities: vicarious trial and error (imagining futures), counterfactual learning (imagining other pasts), and episodic memory (re-simulating real pasts). In AI terms, this is the move from model-free to model-based reinforcement learning. Model-based learning has two hard problems: building a world model and choosing what to simulate.

Frameworks Introduced

• New Ability #1: Vicarious trial and error:
  • What: in the 1930s Edward Tolman saw rats at hard maze forks pause and look back and forth. He guessed they were “playing out” each option, and he called it “vicarious trial and error.”
  • When it happens: only when decisions are hard. That is when costs are close to benefits (a medium food delay, not a short or long one), when the rules change (food moved), or when needs change (the rat is full of one food).
  • Proof: in the 2000s, David Redish and Adam Johnson recorded hippocampal place cells. At the choice point, the cells stopped encoding where the rat was and rapidly played out both possible future paths.
  • Why it works: once an animal has a world model, it can explore it in its head to find a route it was never rewarded for. Failure mode without it: a fish can only “approach” food it sees, so it hits the transparent barrier.
• New Ability #2: Counterfactual learning:
  • What: learning from what would have happened under a different choice. Model-free reinforcement learning can only reinforce the action actually taken.
  • Proof: the “restaurant row” tests and monkeys playing rock, paper, scissors (see the worked example and Mental Models).
  • Link to causation: “X caused Y” means that if X had not occurred, Y would not have occurred. Without counterfactuals you can know only correlations, such as “X always happens before Y.”
• New Ability #3: Episodic memory:
  • What: recalling specific past episodes, as distinct from procedural memory (how to type or throw). Remembering is simulating an approximate re-creation of the past. “Both are simulations.”
  • Evidence: remembering and imagining use the same circuits, and memories are “filled in” just like visual shapes. Eyewitnesses are often 100 percent sure, yet 77 percent of the wrongfully convicted people exonerated by the Innocence Project were convicted on mistaken eyewitness testimony. Repeatedly imagining a false event raises confidence that it happened.
  • Mechanism: the hippocampus learns patterns quickly but cannot simulate, and the neocortex simulates but learns slowly. The hippocampus stores a fast “memory code” that can reactivate a neocortical simulation, as with Henry Molaison (below).

Key Concepts

• Detour task: a fish learns the hole in a transparent barrier but still swims straight at food behind it. A fish that has crossed many times finds the food no faster than one that never crossed. A rat that knows the map gets around much faster.
• Salt-deprivation test: a rat that hated overly salty food runs straight to it once salt-deprived. The path was “vicariously reinforced” before the rat acted. Bennett knows of no fish or reptile that does this.
• Counterfactual: “what the world would be now if you had made a different choice in the past.”
• Credit assignment problem: deciding which earlier events or actions get credit for predicting an outcome.
• Henry Molaison: in September 1953, at 27, his hippocampus was removed to stop seizures. He could no longer form new memories but kept old ones. Forty years later he could do crosswords only with pre-1953 facts.
• Catastrophic forgetting: neural networks forget old patterns when they learn new ones. The hippocampus replays recent memories alongside old ones, which in AI is called “generative replay” or “experience replay.” That is why it is needed to make new memories but not to retrieve old ones.
• Search problem (Marvin Minsky): in most real situations you cannot search all options. Chess has more possible branches than there are atoms in the universe.

The Evolution of Credit Assignment

Early bilaterians	Early vertebrates	Early mammals
Credit assigned by the basic rules of blocking, latent inhibition, and overshadowing	Credit assigned when the critic predicts changes in future rewards	Credit assigned by the counterfactual: which earlier events or actions, had they not occurred, would have prevented the event (what truly caused it)

Model-Free vs Model-Based Reinforcement Learning

	Model-free	Model-based
What it learns	Direct links between the current state and the best actions	A model of how actions affect the world, used to simulate actions before choosing
Trade-off	Faster decisions but less flexible	Slower decisions but more flexible
Emerged in	Early vertebrates	Early mammals
Neocortex	Not required	Required
Example	Going to work by responding to each cue (traffic light, landmark) as it comes up	Considering routes to work and picking the one that was fastest in your mind

Mental Models

• Think of recognition as the means and simulation as the end. Even fish recognize rotated and rescaled objects, so the new thing was imagining.
• Use the pause as a signal. Head-toggling (vicarious trial and error) appears only when a choice is close, new, or changed.
• Think of regret as a learning signal. Rumination was useful in the ancient world because the same situation often came up again.
• Think of causation as something the brain builds because it is useful, not because it can be measured. “Causation is constructed by our brains to enable us to learn vicariously from alternative past choices.”
• Use the rock, paper, scissors test to tell model-free from counterfactual learners. After losing with paper to scissors, a fish-like learner would only avoid paper. A monkey shifts toward rock, the move that would have won.

Anti-patterns

• Assuming the neocortex evolved for recognition, with imagination as a byproduct. The older cortex already recognized well.
• Treating memories as recordings. They are simulations, filled in and easily distorted.
• Dismissing head-turning at a fork as noise. Tolman’s idea faded for decades for lack of evidence, but Redish’s recordings showed rats imagining paths.
• Treating a working world model as enough: model-based RL also needs a way to decide what to simulate. Chess’s model is trivial, but its search is impossible.

Rules of Thumb (author’s wording)

• “Without counterfactuals, there is no way to distinguish between causation and correlation.”
• “When imagining future events, you are simulating a future reality; when remembering past events, you are simulating a past reality.”
• Most RL in technology is model-free: “These systems don’t pause and consider their choices.”

Worked Example

Redish and Steiner’s “restaurant row” test of regret (figure 12.1).

1. Rats run counterclockwise around a circular track past four corridors, each a “restaurant.” The figure labels them chocolate, banana, cherry, and unflavored food pellet.
2. As a rat passes each corridor, a random tone signals the wait if it stays. Tone A means food within forty-five seconds, and tone B means within five seconds.
3. If the rat moves on, it cannot go back without circling the whole track. Each choice is irreversible, and the rat has an hour to get as much favorite food as it can.
4. A rat skips a five-second banana to gamble on its favorite, cherry, and then hears the forty-five-second tone.
5. It pauses and looks back at the passed corridor. Neurons in the taste area of its neocortex reactivate the representation of banana, so it is imagining the choice it did not make.
6. Rats that did this changed later choices: they waited longer next time and ate other food faster to get back around to cherry. That is counterfactual learning changing behavior.

Key Takeaways

1. The neocortex’s first job was probably simulating the world as it is not, not recognizing it.
2. Vicarious trial and error lets mammals solve detours and act on changed needs without new real trials.
3. Counterfactual learning gave a new answer to credit assignment and may be the origin of our sense of causation.
4. Episodic memory is a re-simulation of the past, stored quickly by the hippocampus and rendered by the neocortex.
5. These abilities are model-based reinforcement learning, which is flexible but needs a world model and a solution to the search problem.
6. Outside mammals, such abilities appear mainly in birds (and, for episodic memory, cephalopods), which evolved their own simulating structures.

Connects To

• Ch 4: blocking, latent inhibition and overshadowing, the first credit-assignment rules.
• Ch 6: temporal difference learning, the model-free RL of early vertebrates.
• Ch 9: the hippocampal spatial map of fish, which mammals reuse to play out paths.
• Ch 11: the generative model that renders these simulations.
• Ch 13: how mammals solve the search problem and decide when to simulate.