Chapter 13: Model-Based Reinforcement Learning

Core Idea

AlphaZero beat model-free systems by searching, but only through the moves its actor already favored. Mammals seem to solve the search problem in a similar way, and they add flexibility about when, what, and how finely to simulate. The key is the frontal neocortex. The sensory neocortex renders simulations (a world model), and the agranular prefrontal cortex (aPFC) controls them (a self model). The aPFC learns to explain the animal’s own behavior as “intent.” When its predictions disagree, it pauses the animal and plays out each predicted option. The imagined results then train the basal ganglia. This is the origin of goals, and it also explains attention, working memory, and self-control.

Frameworks Introduced

• AlphaZero’s search: verify the actor’s hunches (DeepMind, 2017):
  • When to use: when exhaustive search is impossible. Simulating every Go move from one position would take over a million years even on Google’s supercomputers.
  • How: take the actor’s top few moves rather than just one, play out thousands of simulated games for each (for example, winning 35 of 40 imagined games with move A and 39 of 40 with move B), and pick the move with the best win ratio. AlphaZero ran about a thousand futures in half a second.
  • Why it works: it is “a clever elaboration on Sutton’s temporal difference learning, not a reinvention of it.” Search checks and extends what the model-free actor already suggests, instead of considering everything.
  • Limits compared with the real world: Go actions are discrete (about two hundred next moves), its information is deterministic and complete, and its reward is simple (win or lose). Real actions are continuous, information is noisy and incomplete, and needs compete and change. AlphaZero also used the same search strategy for every move.
• The three-step choice process (figure 13.4, a speculation):
  • Step 1, triggering simulation (figure label: Uncertainty causes hesitation). aPFC columns are either silent, in agreement, or in disagreement about what the animal will do next. Disagreement is uncertainty, and it triggers a pause through the aPFC’s link to the basal ganglia.
  • Step 2, simulating options (figure label: Options simulated). Only the options the columns already predict get played out (left to water, right to food). The aPFC asks the sensory neocortex “What happens if we go to the left?”, then what happens if it keeps going forward, step by step to the imagined goal. Alternatively, the basal ganglia may pick the actions inside the simulation, which would be even closer to AlphaZero.
  • Step 3, choosing (figure label: Choice made). Each imagined outcome adds votes in the basal ganglia, as real outcomes would, until one passes the choice threshold. The basal ganglia cannot tell imagined from real, so the aPFC has vicariously trained it.
  • Failure mode: without uncertainty there is no pause, which is how habits run even when the goal no longer makes sense.
• Frontal vs sensory neocortex:

	Frontal neocortex	Sensory neocortex
Model	A self model	A world model
Input	Hippocampus, amygdala, and hypothalamus	Sensory organs
Explanation	“I did this because I want to get to water”	“I see this because there is a triangle right there”
Predicts	What the animal will do next	What external objects will do next

Key Concepts

• Frontal neocortex in humans: motor cortex, granular prefrontal cortex (gPFC, six layers) and agranular prefrontal cortex (aPFC, missing layer 4). The gPFC came later in primates, and the motor cortex came after the first mammals. In the first mammals the whole frontal cortex was aPFC (figure 13.3).
• Akinetic mutism: damage to the aPFC leaves people able to move and understand but with no will to act. Damasio’s patient L, in the 1980s, said her mind was “empty” and nothing “mattered.” Others said “she was there but not there.”
• Rats with aPFC damage: they lose vicarious trial and error, episodic recall and counterfactual learning. They plan routes poorly from new start points, take easier paths with fewer rewards, repeat past mistakes, do steps out of order, and become impulsive.
• Uncertainty as disagreement: like ensemble methods in machine learning, where more divergent predictions mean more uncertainty. The aPFC is most active when something goes wrong or something unexpected happens.
• Habits: automated, model-free actions triggered directly by stimuli and run by the basal ganglia. They save time and energy but cause bad choices at the wrong times.
• Intent / goal: a constructed explanation of one’s own behavior, used to predict it. The basal ganglia “has no intent or goals.”
• Active inference (Karl Friston): the sensory cortex does passive inference, explaining input. The aPFC explains behavior and then changes behavior to fit its prediction. That is why it needs little or no layer 4, the layer where sensory input pushes the model to match the data.
• Attention vs imagination: both are the aPFC triggering a simulation. When the simulation is constrained by current input it is attention, and when it is unconstrained it is imagination. Flipping the duck-rabbit is your aPFC at work.
• Working memory: the aPFC re-invoking a simulation until it is no longer needed. Rodents need the aPFC to remember which trees they have already foraged.
• Self-control: the aPFC projects to inhibitory neurons around the amygdala. It costs a lot of energy to run, so tired or stressed people become more impulsive.

Mental Models

• Use disagreement among your predictors as the trigger for slow, deliberate planning. When things go as expected, let the fast model-free system act.
• Think of the aPFC as predicting the animal itself, the way the sensory cortex predicts the world. Intent is to the aPFC what “triangle” is to the visual cortex: a constructed explanation used to predict.
• Think of control as persuasion. “The aPFC controls behavior not by telling but by showing.” It shows the basal ganglia imagined outcomes and filters what information reaches it.
• Think of the basal ganglia and aPFC as teachers who swap roles. The basal ganglia first teaches the aPFC what the animal does, and later the aPFC teaches the basal ganglia. In development, the aPFC starts with a layer 4 that slowly disappears.
• Use one duality across fields:

Field	Simulate first	Act automatically
AI	Model-based	Model-free
Animal psychology	Goal-driven behavior	Habitual behavior
Behavioral economics (Kahneman)	System 2 (thinking slow)	System 1 (thinking fast)

Anti-patterns

• Applying model-free TD learning to deep-planning problems: after TD-Gammon it disappointed at chess, because it cannot find moves that look bad now but pay off later.
• Searching for one special search algorithm: the mammal advantage is more likely flexibility about when to simulate, whether to simulate the future or the past, and at what level of detail.
• Asking a model-free system why it acted: there is no real answer beyond “because it thought that was the choice with the most predicted reward.” This is why such AI is hard to interpret.
• Treating habits as all bad: walking, typing and speaking are habits. The cost comes only when automation fires at the wrong time.

Rules of Thumb (author’s wording)

• “Neither is better; each has its benefits and costs.”
• “Planning, attention, and working memory are all controlled by the aPFC because all three are, in principle, the same thing.”
• The aPFC is “repurposing the neocortical generative model for prediction to create volition.”

Worked Example

Dickinson’s devaluation experiment (Cambridge, early 1980s).

1. Train rats to press a lever for food pellets.
2. Away from the lever box, give them the same pellet laced with a chemical that makes them sick. Afterward they eat far fewer pellets.
3. Put them back at the lever with no pellets delivered. A pure law-of-effect learner would press as before, because nothing has un-reinforced the lever. A simulating rat would picture the pellet and press less.
4. Result: rats with the pellet paired with illness pressed almost 50 percent less, which is evidence of simulating consequences.
5. But it depended on training. Rats that had pressed 100 times stopped. Rats that had pressed 500 times pressed as hard as ever, without ever getting a reward. They were “insensitive to devaluation.”
6. Reading: after 500 repetitions, neither the aPFC nor the basal ganglia detected any uncertainty, so there was no pause to consider consequences. Dickinson had discovered habits.

A second example: lizards versus rats. Lizards need hundreds of trials to learn to go to a red light for good food instead of their hardwired favorite, green. Rats unlearn hardwired responses much faster, but not if their aPFC is damaged.

Key Takeaways

1. Model-based RL works when search is limited to the options the model-free system already suggests.
2. The aPFC models the self. Disagreement among its predictions triggers a pause, simulation, and vicarious training of the basal ganglia.
3. Goals first appeared in early mammals, as aPFC-constructed intent. Model-free systems have none.
4. Habits form when repetition removes uncertainty, so the brain stops pausing, even when the goal no longer applies.
5. Attention, working memory, planning and self-control are all the aPFC choosing which simulation to render.
6. The aPFC is energetically expensive, so tiredness and stress shift control back to the amygdala and basal ganglia.

Connects To

• Ch 6: TD learning, the actor-critic, and TD-Gammon, which AlphaZero builds on.
• Ch 11: the generative model and the duck-rabbit illusion. Layer 4 and the missing granule cells come from the same microcircuit.
• Ch 12: vicarious trial and error, counterfactual learning and the search problem that this chapter answers.
• Ch 14: the motor cortex runs the same predict-then-fulfill trick for body movements.
• Ch 15–18: the granular prefrontal cortex of primates and modeling one’s own mind.