Chapter 4: Associating, Predicting, and the Dawn of Learning

Core Idea

Associative learning, the kind Pavlov found in his salivating dogs, first appeared in the early bilaterians. Its job was to adjust steering by changing what counts as good and bad based on experience. Radially symmetric animals such as anemones never learned associations because a stationary animal gains little from them. A steering animal gains a lot. Simple brains had to solve two hard problems that AI still faces. The first is the continual learning problem, which is learning while the rules of the world keep changing. The second is the credit assignment problem, which is deciding which earlier cue predicted an outcome. They solved these with crude but effective tricks built into synapses. Every later form of learning reused the same synaptic mechanisms.

Frameworks Introduced

• The four stages of a changing association (continual learning):

  • When to use: when a learner must keep learning after training, in a world whose rules shift.
  • How it works:
    • Acquisition: each pairing (buzzer then food) strengthens the association.
    • Extinction: each cue without the outcome weakens it.
    • Spontaneous recovery: an extinguished association returns after a few days. It was suppressed, not unlearned.
    • Reacquisition: an old, extinguished association is relearned much faster than a new one.
  • Figure 4.2 plots saliva produced to a bell over time. Saliva rises along a curve during acquisition. It falls off once the bell comes without food. Then it spikes briefly when the bell is tried again several days later.
  • Why it works: one failure may be a fluke. Spontaneous recovery keeps a long-term memory through short-term changes. Reacquisition lets old rules return quickly when the world shows signs that they apply again.

• The original four tricks for the credit assignment problem (the book’s table):

  Trick	Rule
  Eligibility traces	Pick the predictive cue that occurred between 0 to 1 second before the event.
  Overshadowing	Pick the predictive cue that was the strongest.
  Latent inhibition	Pick the predictive cue that you haven’t seen before.
  Blocking	Stick to predictive cues once you have them and ignore others.

  • Why it works: each trick is a cheap rule of thumb that is usually right. Useful cues tend to come just before the event. Strong cues are more reliable. New things are the likely cause of new outcomes. And redundant cues add nothing.
  • Failure mode: the tricks are imperfect. Sometimes the best cue came a minute earlier, or the best cue is the weak one. Breakthroughs #2 and #3 add better solutions.

• Hebbian learning (the synaptic mechanism): protein machinery at a synapse detects whether the input neuron and output neuron fired in the same time window, and if so it strengthens the connection. It is often called the rule that “neurons that fire together wire together.” Extra synaptic machinery adds timing (input must fire just before output, which gives eligibility traces). Dopamine or serotonin can gate learning, because some synapses learn only when those receptors are also active.

Key Concepts

• Conditional vs unconditional reflex: Pavlov’s terms. An unconditional reflex is innate (salivating to sugar in the mouth). A conditional reflex depends on a prior association (salivating to a buzzer that preceded food).
• Psychic stimulation: Pavlov first saw dogs salivating before they tasted food as a “source of error”. It turned out to be associative learning.
• Involuntary learning: conditional reflexes happen without conscious involvement. Even a rat with its brain removed learns to pull back its tail when a leg tap is paired with a tail tap, using only spinal cord circuits.
• Valence can be learned: experience can turn a good stimulus bad, or a bad one good.
• The continual learning problem: as of 2023, most AI systems are trained once and then used without further learning. If the world changes in ways the training data did not cover, they must be retrained or they make catastrophic mistakes. Biological brains have always learned continually.
• The credit assignment problem: a machine learning term. When something happens, which earlier cue gets credit for predicting it?
• Eligibility trace: the short window after a stimulus during which it can be associated. In slugs, a tap one second before a shock is associated with it, but a tap two seconds or more before is not.
• Latent inhibition: frequent stimuli are flagged as background noise. It asks, “What was different this time?”
• Synaptic learning: learning means synapses getting stronger or weaker, or synapses being added or removed. It is not Plato’s impressions in wax, Descartes’s “folds”, or “vibrations”. A strong synapse needs few input spikes to make the output neuron fire, and a weak one needs many (figure 4.3).

Mental Models

• Think of learning in the first brains as “merely a feature, a trick to optimize steering decisions”, not the brain’s core function. The rest of the book describes learning becoming the core function.
• Use spontaneous recovery as a model for weighing recent evidence against a long history. Suppress the old rule after a failure, but do not delete it.
• Think of the credit assignment tricks as heuristics for deciding what caused an outcome. They are timing, strength, novelty, and sticking with what already works.
• Use the coral-versus-worm contrast to predict when learning pays off. If your strategy is to sit and wait, hardcoded reflexes are enough. If you choose where to go, experience changes what you should choose.
• Think of later learning (spatial maps, language, object recognition, music) as new uses of the same synaptic mechanisms, not new mechanisms.

Anti-patterns

• Assuming learning requires advanced brain structures: associative learning works in spinal cord circuits and in 302-neuron nematodes.
• Treating extinction as forgetting: extinguished associations come back (spontaneous recovery) and are relearned fast (reacquisition).
• Deploying a frozen model into a changing world: if a law required driving on the left, a car trained only on the right could not adjust without explicit retraining.
• Overrating what early learning could do: early bilaterians could not link events more than a few seconds apart, predict exact timing, recognize objects or patterns, or recognize locations or directions.

Rules of Thumb (author’s wording)

• “Once animals began approaching specific things and avoiding others, the ability to tweak what was considered good and bad became a matter of life and death.”
• “stimuli that are useful for predicting things should occur right before the thing you are trying to predict.”
• “Blocking is a way to stick to one predictive cue and avoid redundant associations.”
• “Association, prediction, and learning emerged for tweaking the goodness and badness of things.”

Worked Example

Salt becomes bad: the nematode hunger experiment (figure 4.1).

1. Take a hundred nematodes. Put half in a dish of plain water and half in a dish of salty water. Neither dish has food, so after several hours both groups are uncomfortably hungry.
2. Move both groups to a dish with a small piece of salt on one side.
3. The plain-water group steers toward the salt, as nematodes normally do, since salt usually has positive valence.
4. The salt-water group steers away from the salt. Salt has become negative because it was paired with the negative state of hunger.
5. Figure 4.1 also shows a control group kept with salt and food and no hunger. That group still steers toward the salt, so the change comes from pairing salt with hunger, not from exposure to salt alone.
6. The same kind of learning appears across bilaterians. A food smell paired with a chemical that makes nematodes sick becomes something to avoid. Feeding at a specific temperature shifts preference toward it. A slug learns to withdraw from a tap paired with a shock, and the association lasts for days. An anemone never learns to withdraw from a light paired with shock, no matter how many pairings.
7. Continual learning in the wild: if the worm later steers to salt and finds no food for an hour, the association is extinguished. If it smells salt again two days later, the better choice is to approach again, because most past experiences linked salt to food. That is spontaneous recovery. After twenty more failures, the association may be extinguished for good.

Key Takeaways

1. Associative learning appeared together with valence in the first bilaterians, about 550 million years ago, to change what is good and bad.
2. Learning in biological brains has always been continual. Modern AI as of 2023 mostly is not.
3. Acquisition, extinction, spontaneous recovery, and reacquisition let simple brains follow rules that change.
4. Eligibility traces, overshadowing, latent inhibition, and blocking are crude but working answers to the credit assignment problem. They are found across bilaterians, from flatworms to rats.
5. Learning is physical. It happens through changes in synapses, through Hebbian learning that neuromodulators can gate.
6. Non-bilaterians such as coral polyps lack some of the proteins involved in Hebbian learning, which is consistent with associative learning starting in bilaterians.

Connects To

• Ch 2: learning adjusts the valence that steering depends on.
• Ch 3: dopamine released by food can trigger the strengthening of synapses, so affect and learning are linked.
• Ch 5–9 (Breakthrough #2: Reinforcing): better solutions to credit assignment and continual learning, with learning becoming central to the brain’s function.
• Ch 10–14 (Breakthrough #3: Simulating): further strategies for credit assignment.

Summary of Breakthrough #1: Steering

About 550 million years ago our ancestors changed from radially symmetric, brainless animals like coral polyps into bilaterally symmetric animals with brains, like nematodes. Many of the changes in this transition can be understood as serving one breakthrough, navigating by steering. Breakthrough #1 added these:

• A bilateral body plan reduced navigation to two choices: go forward or turn.
• A neural architecture for valence hard-coded stimuli as good or bad.
• Mechanisms let internal states change valence responses.
• Circuits combined different valence neurons into one steering decision. This large cluster of neurons is what we call a brain.
• Affective states made lasting decisions about whether to leave or stay.
• The stress response managed the energy cost of movement during hardship.
• Associative learning changed steering decisions based on previous experience.
• Spontaneous recovery and reacquisition dealt with changing rules in the world, making continual learning work, even if imperfectly.
• Eligibility traces, overshadowing, latent inhibition, and blocking tackled the credit assignment problem, though imperfectly.

These changes made our ancestors the first large multicellular animals to survive by navigating with muscles and neurons rather than cellular propellers. They, and the predatory ecosystem they produced, laid the foundation for breakthrough #2, when learning became central to the brain’s function.