Abstract
Replay can consolidate memories by offline neural reactivation related to past experiences. Category knowledge is learned across multiple experiences and subsequently generalised to new situations. This ability to generalise is promoted by offline consolidation and replay during rest and sleep. However, aspects of replay are difficult to determine from neuroimaging studies alone. Here, we provide a comprehensive account of how category replay may work in the brain by simulating these processes in a neural network which assumed the functional roles of the human ventral visual stream and hippocampus. We showed that generative replay, akin to imagining entirely new instances of a category, facilitated generalisation to new experiences. This invites a reconsideration of the nature of replay more generally, and suggests that replay helps to prepare us for the future as much as remember the past. We simulated generative replay at different network locations finding it was most effective in later layers equivalent to the lateral occipital cortex, and less effective in layers corresponding to early visual cortex, thus drawing a distinction between the observation of replay in the brain and its relevance to consolidation. We modelled long-term memory consolidation in humans and found that category replay is most beneficial for newly acquired knowledge, at a time when generalisation is still poor, a finding which suggests replay helps us adapt to changes in our environment. Finally, we present a novel mechanism for the frequent observation that the brain selectively consolidates weaker information, and showed that a reinforcement learning process in which categories were replayed according to their contribution to network performance explains this well-documented phenomenon, thus reconceptualising replay as an active rather than passive process.
Author Summary The brain relives past experiences during rest. This process is called “replay” and helps strengthen our memories and promote generalisation. We learn over time to categorise objects, yet how category knowledge is replayed in the brain is not well understood. We used a neural network which behaves like the human visual brain to simulate category replay. We found that allowing the network to “dream” typical examples of a category during “night-time” consolidation was an effective form of replay that helped subsequent recognition of unseen objects, offering a solution for how the human brain consolidates category knowledge. We also found this to be more effective if it took place in advanced layers of the network, suggesting human replay might be most effective in high-level visual brain regions. We also tasked the network with learning to control its own replay, and found it focused on categories that were difficult to learn. This represents the first mechanistic account of why weakly-learned memories in humans show the greatest improvement after rest and sleep. Our approach makes predictions about category replay in the human brain which can inform future experiments, and highlights the value of large-scale neural networks in addressing neuroscientific questions.
Competing Interest Statement
The authors have declared no competing interest.
