Chapter 6: The Evolution of Temporal Difference Learning

Core Idea

Learning only from actual rewards fails, because a reward at the end of a long sequence does not say which earlier actions earned it. Richard Sutton solved this with temporal difference (TD) learning. A critic predicts future reward at every moment, and an actor is reinforced by changes in that prediction rather than by the reward itself. Dopamine recordings later showed that vertebrate brains compute the same signal. Bennett proposes that the basal ganglia is the actor-critic system, with the hypothalamus deciding actual rewards. Disappointment, relief, and precise timing follow from learning by predicted reward.

Frameworks Introduced

• The temporal credit assignment problem:
  • What it is: when the reward comes only at the end (winning or losing a game of hundreds of moves), which earlier moves deserve credit or blame? The bilaterian tricks from chapter 4 (overshadowing, latent inhibition, blocking) handle stimuli that overlap in time, not ones separated in time. Minsky identified the problem as far back as 1961.
  • Failed fix 1: reinforce the moves just before the outcome, giving less credit the longer ago a move was. This is how Minsky’s SNARC worked. It gives the last moves most of the credit, even when a single clever early move won the game.
  • Failed fix 2: reinforce or punish every move in the game equally and hope enough games sort them out. There are too many possible games: over five hundred quintillion in checkers and over 10^120 in chess.
• Temporal difference learning (the actor and the critic):
  • When to use: any task where reward is delayed and sparse relative to the actions that cause it.
  • How: split the learner in two. The critic predicts the likelihood of winning from the current state. The actor chooses actions and is reinforced whenever the critic’s prediction goes up after its move, and punished when it goes down. The learning signal is “the temporal difference in the predicted reward from one moment in time to the next.”
  • Figure 6.1 shows a checkers board (the state) feeding two boxes. The actor outputs a “Predicted best next move.” The critic outputs a number such as “64% chance of winning.”
  • Figure 6.2 compares two designs. On the left, reinforcement based on actual rewards gives learning only at the end, from the win back to earlier moves. On the right, a critic sits beside every move and sends learning to the actor at each step, labeled “Learning continuously based on “temporal difference” in predicted likelihood of winning.”
  • Why it works: actor and critic train each other. The logic looks circular, since “the critic depends on the actor, and the actor depends on the critic.” But in Sutton’s simulations “a magical bootstrapping” occurred, and both improved until they converged.
  • Failure mode: early in training the critic often rewards the wrong actions, and the actor often fails to carry out what the critic predicts. It takes many games to converge.
• Dopamine as the TD signal:
  • Dopamine bursts when predicted future reward rises (a positive temporal difference). It stays at baseline when an expected reward arrives (no difference). It dips when an expected reward is omitted (a negative difference).
  • It also tracks discounting and probability. A cue predicting food in four seconds triggers more dopamine than one predicting food in sixteen seconds. A 75 percent cue triggers more than a 25 percent cue.
• The first vertebrate brain design (Figure 6.4):
  • Sensory neurons feed both the cortex and the hypothalamus. The cortex sends “Info about external world” to the basal ganglia and to the motor system. Hypothalamic valence neurons drive dopamine neurons, which send dopamine to the basal ganglia. The basal ganglia sends a signal back to the dopamine neurons, and it “Gates & ungates specific motor programs” in the midbrain and hindbrain through inhibitory connections.
  • Roles: the hypothalamus is the judge of actual rewards, such as food when hungry or warmth when cold. It tells the brain whether it won the game, not how the game is going. Within the basal ganglia, parallel circuits act as the actor (gating movement) and the critic (predicting reward and triggering dopamine). Bennett presents this as one leading theory, not a settled fact.

Key Concepts

• SNARC: Marvin Minsky’s 1951 Stochastic Neural-Analog Reinforcement Calculator. It had forty connections and strengthened recently active synapses after escaping a maze. It failed on anything slightly complex.
• TD-Gammon: Gerald Tesauro’s backgammon system at IBM Research, built in the early 1990s. It learned only by trial and error, unlike the earlier Neurogammon, which copied expert games and stalled at an intermediate level. By 1994 it played as well as some of the best players in the world.
• Discounting: valuing sooner rewards above later ones. It pushes agents and animals toward faster rewards.
• Reward vs reinforcement: “Dopamine is not a signal for reward but for reinforcement.” Actual reward and the learning signal must be separated.
• Basal ganglia: a structure between the cortex and the thalamus whose output nucleus inhibits brainstem motor circuits by default. Actions happen only when specific basal ganglia neurons turn off and so ungate them. It is nearly identical in humans and lampreys.
• Hypothalamus: the vertebrate home of the valence neurons inherited from early bilaterians. Its responses depend on internal state, so pizza is wanted when you are hungry and rejected when you are full.
• Disappointment and relief: the omission of an expected reward punishes, and the omission of an expected punishment reinforces.
• Avoidance task: a fish learns to swim away after a light to avoid a shock that comes five seconds later. Here the reinforcer is something that did not happen.
• Interval timing: vertebrates can learn that one event follows another by exactly five seconds. Slugs cannot associate events more than two seconds apart, and crabs and bees cannot learn precise intervals.

Mental Models

• Use predicted reward, not actual reward, as the training signal whenever the outcome is far from the actions that caused it.
• Think of the basal ganglia as a student who first learns from a “vague but stern hypothalamic judge” and then learns to judge itself before the judge gives feedback.
• Think of a fully expected reward as producing no learning signal, because the critic’s prediction cancels the hypothalamus’s response.
• Use timing as a prerequisite for learning from omission. Without a clock, a brain “cannot know whether something was omitted or simply hasn’t happened yet.”
• Think of dopamine’s evolution as a repurposing: from a fuzzy “good-things-are-nearby” signal in bilaterians to a precise signal of predicted future reward.

Anti-patterns

• Dopamine as pleasure: dopamine is about wanting, not liking (chapter 3), and its responses fit TD learning, not pleasure.
• Dopamine as surprise: an omitted reward is just as surprising as a delivered one, yet dopamine drops instead of rising.
• Rewarding only the last steps or all steps equally: both fail on long tasks, as shown above.
• Learning from experts only: Neurogammon imitated experts and plateaued, while TD-Gammon, learning by trial and error, surpassed it.

Rules of Thumb (author’s wording)

• “Instead of rewarding an AI system when it wins, what if you reward it when the AI system thinks it is winning?”
• “Dopamine is not a signal for reward but for reinforcement.”
• “The basal ganglia learns to repeat actions that maximize dopamine release.”
• “Sutton discovered a trick that evolution had already stumbled upon over five hundred million years ago.”

Worked Example

Schultz’s dopamine recordings, decoded by Dayan and Montague.

1. In the 1980s Wolfram Schultz became the first to record single dopamine neurons. He showed monkeys a cue, such as a geometric shape, and a few seconds later gave them sugar water.
2. On the first trial, dopamine neurons burst when the sugar water arrived (Figure 6.3, first row, five seconds after the cue).
3. After a few trials, the burst moved to the cue. When the expected sugar water then arrived, activity stayed at baseline (second row).
4. When Schultz showed the cue but gave no sugar water, activity paused at the exact moment the reward should have come (third row). This ruled out “dopamine as surprise,” because the omission was just as surprising as an unexpected reward.
5. About a decade later, Peter Dayan (one of Sutton’s students) and Read Montague saw that each response matched Sutton’s TD signal. The cue raises predicted reward, giving a positive difference. The expected reward does not change the prediction, giving no difference. The omission lowers it, giving a negative difference.
6. In 1997 they published “A Neural Substrate of Prediction and Reward” with Schultz. Bennett calls it one of the most famous partnerships between AI and neuroscience: an AI method inspired by a guess about the brain then explained puzzling brain data.
7. The same TD signals appear in fish, rat, monkey, and human brains, but not in the dopamine neurons of nematodes or other simple bilaterians.

Key Takeaways

1. The temporal credit assignment problem blocked reinforcement learning for decades, and neither rewarding the recent moves nor rewarding all moves solves it.
2. TD learning reinforces an actor using changes in a critic’s predicted reward, and the two improve each other through bootstrapping.
3. Dopamine in vertebrates carries a TD signal: it bursts for gains in predicted reward, stays flat for expected reward, and dips for omissions.
4. The basal ganglia gates actions and learns to repeat what releases dopamine, while the hypothalamus supplies the actual rewards.
5. Disappointment, relief, learning from omission, and precise timing all come with TD learning, and invertebrates such as nematodes, crabs, and bees lack them.

Connects To

• Ch 3: dopamine as wanting rather than liking.
• Ch 4: the credit assignment tricks of bilaterians, which work only for cues that overlap in time.
• Ch 5: Thorndike’s law of effect, which TD learning makes workable.
• Ch 8: TD learning needs curiosity to explore enough.
• Ch 13: model-based reinforcement learning in mammals extends the model-free TD learning described here.