Chapter 8: Why Life Got Curious

Core Idea

Reinforcement learning needs trials, so an agent must explore new behaviors as well as exploit known ones. Random exploration fails when the first reward is far away, as in the Atari game Montezuma’s Revenge. Making surprise itself reinforcing, which is curiosity, solved that game in 2018. Vertebrates have the same mechanism, because surprise releases dopamine even without a real reward. The same mechanism explains why gambling and social media feeds are hard to stop.

Frameworks Introduced

• The exploitation-exploration dilemma (called the exploration-exploitation dilemma in the part summary):
  • What it is: reinforcement learning needs two opposing processes. Exploitation repeats behaviors that were reinforced before, and exploration tries new ones. Each choice works against the other.
  • Crude solution: early TD learning systems acted at random a fixed share of the time, “say, 5 percent of the time.” This worked in games with few possible moves and failed in games with nearly endless places to go.
• Curiosity as intrinsic reward:
  • When to use: when external rewards are sparse or delayed and the space of actions is large.
  • How: reward the agent for novelty and surprise, so that the greater the novelty, the stronger the pull to explore it. The agent then explores on purpose, seeking new rooms and new actions, instead of acting randomly.
  • Why it works: progress becomes rewarding before any external reward appears. Reaching a new room is valuable in itself.
  • Failure mode: if surprise is reinforcing in itself, an activity with negative expected reward can still be pursued when it is novel. Gambling and social feeds use this.

Key Concepts

• Montezuma’s Revenge: an Atari game in which the first reward comes only after finding a hidden door in a faraway hidden room. TD learning beat humans at Pinball, Star Gunner, Robotank, Road Runner, Pong, and Space Invaders, but not at this game.
• Curiosity in vertebrates: seen in fish, mice, monkeys, and human infants. Among invertebrates, only advanced ones such as insects and cephalopods show it, and it evolved separately in them.
• Variable-ratio reinforcement: food delivered after a random number of lever presses. B. F. Skinner found that rats press far more under it than under fixed-ratio reinforcement, even when the total number of pellets is the same. Fish show the effect too.
• Gambling’s design: the chance of winning is about 48 percent. That is high enough to seem possible, uncertain enough that a win is surprising and gives a dopamine boost, and low enough that the casino wins in the long run.

Mental Models

• Think of curiosity as a requirement of reinforcement learning, not an extra. In Bennett’s words, “Curiosity and reinforcement learning coevolved because curiosity is a requirement for reinforcement learning to work.”
• Use novelty bonuses instead of random actions when the action space is large.
• Think of slot machines and endless feeds as exploiting a 500-million-year-old preference for surprise. They create “a maladaptive edge case that evolution has not had time to account for.”
• Think of learning itself as becoming valuable once a brain can recognize patterns and remember places. The more it explored, the more links between actions and outcomes it could learn.

Anti-patterns

• Exploring by fixed-rate randomness in open-ended tasks: it fails when there are almost infinite directions to try.
• Assuming behavior always follows expected value: gamblers keep gambling despite a negative expected reward, which goes against Thorndike’s law of effect.
• Treating feed scrolling as a free choice: the behavior is reinforced below awareness, the way gamblers and addicts keep going against their own wishes.

Rules of Thumb (author’s wording)

• “Curiosity and reinforcement learning coevolved because curiosity is a requirement for reinforcement learning to work.”
• “In vertebrates, surprise itself triggers the release of dopamine, even if there is no “real” reward.”
• For the first vertebrates, “learning became, in and of itself, an extremely valuable activity.”
• “Even though there are no explicit rewards until you get past all the rooms in level one, these AI systems didn’t need any external rewards to explore.”

Worked Example

Curiosity beats Montezuma’s Revenge.

1. After TD-Gammon, TD learning systems beat human-level play in many Atari games, but Montezuma’s Revenge stayed out of reach.
2. The first reward comes so late that nothing earlier gets reinforced or punished, so a TD learner has no signal to follow.
3. Random exploration (acting at random a small share of the time) cannot cover a map with practically endless directions.
4. In 2018 Google’s DeepMind gave the agent an intrinsic reward for novelty, which made surprise reinforcing.
5. The agent began to act more like a human player. It went into new rooms and spread across the map on purpose.
6. With no external reward until the end of level one, it still kept making progress, and it eventually completed level one.

Key Takeaways

1. Reinforcement learning needs both exploitation and exploration, and the two conflict.
2. Random exploration works only in small action spaces. Curiosity, meaning reward for novelty, works in large ones.
3. Vertebrates, from fish onward, release dopamine for surprise alone, which suggests that curiosity arose with reinforcement learning in early vertebrates.
4. The same mechanism makes variable rewards, gambling, and social feeds strongly reinforcing.

Connects To

• Ch 6: TD learning and dopamine, which curiosity adds to.
• Ch 7 and 9: pattern recognition and spatial maps made exploring worth the effort.
• Ch 5: Thorndike’s law of effect, which gambling violates.