Abstract
Problem-solving and reasoning involve mental exploration and navigation in sparse relational spaces. A physical analogue is spatial navigation in structured environments such as a network of burrows. Recent experiments with mice navigating a labyrinth show a sharp discontinuity during learning, corresponding to a distinct moment of ‘sudden insight’ when mice figure out long, direct paths to the goal. This discontinuity is seemingly at odds with reinforcement learning (RL), which involves a gradual build-up of a value signal during learning. Here, we show that biologically-plausible RL rules combined with persistent exploration generically exhibit discontinuous learning. In structured environments, positive feedback from learning generates a traveling ‘reinforcement wave’. The discontinuity occurs when the wave reaches the starting point. Task difficulty and the learning rule alter its profile and speed, which are determined by the nonlinear dynamics between the nose and bulk of the wave. Predictions explain existing data and motivate specific experiments to isolate the phenomenon. Additionally, we characterize the exact learning dynamics of various RL rules for a complex sequential task.
Competing Interest Statement
The authors have declared no competing interest.
