Can neurogenesis act as a neural regularizer?

New neurons are continuously generated in the subgranular zone of the dentate gyrus throughout adulthood. These new neurons gradually integrate into hippocampal circuits, forming new naïve synapses. Viewed from this perspective, these new neurons may represent a significant source of ‘wiring’ noise in hippocampal networks. In machine learning, such noise injection is commonly used as a regularization technique. Regularization techniques help prevent overfitting training data, and allow models to generalize learning to new, unseen data. Using a computational modeling approach, here we ask whether a neurogenesis-like process similarly acts as a regularizer, facilitating generalization in a category learning task. In a convolutional neural network (CNN) trained on the CIFAR-10 object recognition dataset, we modeled neurogenesis as a replacement/turnover mechanism, where weights for a randomly chosen small subset of neurons in a chosen hidden layer were re-initialized to new values as the model learned to categorize 10 different classes of objects. We found that neurogenesis enhanced generalization on unseen test data compared to networks with no neurogenesis. Moreover, neurogenic networks either outperformed or performed similarly to networks with conventional noise injection (i.e., dropout, weight decay, and neural noise). These results suggest that neurogenesis can enhance generalization in hippocampal learning through noise-injection, expanding on the roles that neurogenesis may have in cognition. Author Summary In deep neural networks, various forms of noise injection are used as regularization techniques to prevent overfitting and promote generalization on unseen test data. Here, we were interested in whether adult neurogenesis– the lifelong production of new neurons in the hippocampus– might similarly function as a regularizer in the brain. We explored this question computationally, assessing whether implementing a neurogenesis-like process in a hidden layer within a convolutional neural network trained in a category learning task would prevent overfitting and promote generalization. We found that neurogenesis regularization was as least as effective as, or more effective than, conventional regularizers (i.e., dropout, weight decay and neural noise) in improving model performance. These results suggest that optimal levels of hippocampal neurogenesis may improve memory-guided decision making by preventing overfitting, thereby promoting the formation of more generalized memories that can be applied in a broader range of circumstances. We outline how these predictions may be evaluated behaviorally in rodents with altered hippocampal neurogenesis.

Regularization techniques help prevent overfitting training data, and allow models to generalize learning to new, unseen data. Using a computational modeling approach, here we ask whether a neurogenesis-like process similarly acts as a regularizer, facilitating generalization in a category learning task. In a convolutional neural network (CNN) trained on the CIFAR-10 object recognition dataset, we modeled neurogenesis as a replacement/turnover mechanism, where weights for a randomly chosen small subset of neurons in a chosen hidden layer were re-initialized to new values as the model learned to categorize 10 different classes of objects. We found that neurogenesis enhanced generalization on unseen test data compared to networks with no neurogenesis. Moreover, neurogenic networks either outperformed or performed similarly to networks with conventional noise injection (i.e., dropout, weight decay, and neural noise). These results suggest that neurogenesis can enhance generalization in hippocampal learning through noise-injection, expanding on the roles that neurogenesis may have in cognition.

Author Summary
In deep neural networks, various forms of noise injection are used as regularization techniques to prevent overfitting and promote generalization on unseen test data. Here, we were interested in whether adult neurogenesis-the lifelong production of new neurons in the hippocampus-might similarly function as a regularizer in the brain. We explored this question computationally, assessing whether implementing a neurogenesis-like process in a hidden layer within a convolutional neural network trained in a category learning task would prevent overfitting and promote generalization. We found that neurogenesis regularization was as least as effective as, or more effective than, conventional regularizers (i.e., dropout, weight decay and neural noise) in improving model performance. These results suggest that optimal levels of Introduction Noise reflects random or unpredictable fluctuations that are not part of a signal (Faisal et al 2008). Within the brain there are multiple sources of noise, including processes at the cellular (e.g., protein production and degradation), electrical (e.g., membrane potential) or synaptic levels (e.g., vesicular release) that collectively impact the probability and timing of action potentials (Faisal et al., 2008;McDonnell & Ward, 2011).
While neural noise is often considered an obstacle in extracting relevant information from the brain's output activity, optimal levels of neural noise (encapsulated in a broad range of phenomena termed stochastic facilitation (McDonnell & Ward, 2011)) may enhance information transmission and behavior. Similarly, in machine learning noise can also be used to achieve better performance. One of the most common examples is that the addition of some optimized level of noise enhances a model's ability to avoid overfitting and enhances generalization (i.e., avoiding the memorization of training data that is not beneficial to the transfer of learning to unseen data) (Hinton & Camp, 1993).
This process of preventing overfitting in order to enhance generalization on unseen data is termed regularization. We, and others, have suggested that neural noise may be one such strategy by which the brain performs regularization to better extract the statistical regularities of our experiences (Hoel, 2021;Richards & Frankland, 2017).
In the hippocampus, another potential source of neural noise is ongoing neurogenesis.
Asymmetric division of neural precursor cells in the subgranular zone gives rise to newborn neurons that synaptically integrate into hippocampal networks throughout life (Abrous et al., 2005;Denoth-Lippuner & Jessberger, 2021;Gonçalves et al., 2016;Ming & Song, 2005). This process may be considered a form of noise injection in two respects.
First, these new neurons are naive and therefore do not encode any aspects of past experience. Second, their integration gradually reconfigures hippocampal networks. Therefore, we hypothesize that this form of 'wiring' noise may function to regularize learning in the hippocampus, and here we will explore these ideas computationally using categorical learning tasks.
To study whether neurogenesis can act as a regularizer, we implemented neurogenesis in hidden layers of a traditional deep learning architecture (i.e., a convolutional neural network [CNN]). In adult rodents, loss of developmentally-generated granule cells is balanced by the addition of new neurons (Ciric et al., 2019;Cole et al., 2020). Therefore, we modeled neurogenesis as a "replacement/turnover" mechanism, where a randomly chosen small subset of neurons in the middle layer is "turned over" such that their input and output weights are re-initialized to new values (whereas connections of mature neurons remain the same). This turnover/replacement model ensures that the size of the network remains constant. From a computational perspective, this controls for the fact that networks of different sizes can perform very differently (since larger networks have more tunable parameters). From a biological perspective, this mimics the rodent hippocampus where neurogenesis produces negligible net growth (i.e., increase in total number of granule cells) during adulthood (Cole et al., 2020). While deep learning models with neurogenesis have been previously developed (Aimone, 2016;Draelos et al., 2017), these did not explicitly evaluate the role of neurogenesis in generalization.

Neurogenesis improves generalization in CNNs
We implemented neurogenesis in a CNN trained and tested on the CIFAR-10 dataset (Krizhevsky, 2009). CIFAR-10 consists of 60,000 32 ⨉ 32 pixel RGB images in 10 different classes representing airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. The CNN consists of 64, 64, and 128 filters, respectively (see Methods for more information), followed by three fully connected layers ( Figure 1A-B). In order to model neurogenesis, weights of a randomly selected subpopulation of neurons in the middle layer of the fully-connected layers were reinitialized continuously during training, with multiple turnover events. Following hyperparameter tuning, we found that turning over 3.2% of neurons every 640 mini-batch updates to the model had the greatest impact on performance of the network ( Figure 1C). Neurogenesis was restricted to only one layer, mimicking the occurrence of neurogenesis only in the dentate gyrus layer of the hippocampus. We found that neurogenesis in the second fully connected output layer of the network performed the best during hyperparameter tuning, and therefore used this configuration for all experiments. The performance of neural networks is also dependent on initialization (with some random initializations outperforming others even after undergoing the same training process). To control for potential differences in initialization states between groups, each network initialization was copied and these identical versions were used in the control (no neurogenesis) and experimental (neurogenesis) networks, respectively.
We found that neurogenesis improved network performance on unseen test data.
Whereas test accuracy for the control network was 74.36 ± 0.16 %, networks trained with neurogenesis had a test score of 76.20 ± 0.20 % (Figure 2A). This improvement in performance with neurogenesis did not depend on initialization states. In particular, the performance-enhancing effects of neurogenesis were not limited to low-scoring variants. Higher-scoring variants also benefited from neurogenesis, indicating that neurogenesis can improve generalization beyond the top scores that a static, nonneurogenic network can achieve ( Figure 2B).
Adult-generated neurons are more excitable than their developmentally-generated counterparts, with excitability peaking between 4-8 weeks of cell age (Dieni et al., 2016;Doetsch & Hen, 2005;Marín-Burgin et al., 2012;Mongiat et al., 2009;Schmidt-Hieber et al., 2004). To address whether elevated excitability of new neurons might further promote regularization and improve the performance of neurogenic networks, we increased the activation of new neurons by 30% during each forward pass during training ( Figure 2C). To reflect the transient nature of these changes, following each round of new neuron turnover, excitability of previously turned over neurons returned to baseline. As before, we found that neurogenic networks outperformed our control networks (i.e., non-neurogenic networks) on held-back test data. However, incorporating excitability into the neurogenic CNN did not further improve performance (i.e., neurogenesis + excitability) ( Figure 2D).

Neurogenesis improves CNN performance via regularization
The improved performance on held-back test data might reflect a more powerful model in the neurogenesis group, rather than a regularization effect. In other words, it could be that learning in general is simply better in the neurogenesis networks, in the same way that networks with more layers tend to learn better. Indeed, in adult rodent studies interventions that elevate hippocampal neurogenesis facilitate learning in many (Creer et al., 2010;Sahay, Scobie, et al., 2011) situations. (Though it should be noted that this does not actually hold in all situations (Frankland, 2013)). Similarly, in artificial neural networks, implementation of neurogenesis may improve learning in general (Chambers et al., 2004;Deisseroth et al., 2004;Meltzer et al., 2005), rather than having a specific effect on generalization. To assess this, we compared training accuracy of the control and neurogenic networks, anticipating that training accuracy would be the same, or even reduced, in the neurogenesis network should improved performance be due to regularization. We found that the neurogenic networks do not improve training accuracy, and in fact, they sacrifice training accuracy in order to achieve the previously observed higher test accuracy on held-back data. Neurogenic networks achieve lower validation loss, but higher training loss relative to control, suggesting that the enhancement in performance is indeed due to a regularization effect (Figure 2E,F).

Neurogenesis regularization achieves similar level improvement to dropout
We next compared neurogenesis to conventional regularization methods, including dropout, weight decay, and neural noise ( Figure 3A-C). Dropout involves the stochastic silencing of a subset of units during each forward pass (Srivastava et al., 2014). Weight decay involves adding a small penalty to the loss function that penalizes large weights, thus resulting in an overall decay of larger weights (Krogh & Hertz, 1992). Neural noise involves the addition of Gaussian noise to all the activations during each forward pass (Bishop, 1995). After identifying the optimal parameter values for each of these regularization methods, we compared the performance of neurogenic networks to networks regularized with these other methods. We found that neurogenic CNNs performed similarly to dropout, and outperformed weight decay and neural noise ( Figure 3D).
Mechanistically, there are similarities between replacement/turnover neurogenesis and dropout, with both involving information loss at each update. In dropout, a subpopulation of neurons is transiently silenced, but connection weights are otherwise maintained. In contrast, replacement/turnover neurogenesis involves resetting the weights of a subpopulation of neurons. Interestingly, the optimal size of these subpopulations differs markedly for dropout vs. neurogenesis. Optimal performance occurred when dropout was implemented in a subpopulation that corresponded to ~20% of hidden layer neurons, whereas for neurogenesis, optimal performance was achieved when the turnover subpopulation was ~3.2% of hidden layer neurons. We next asked whether combining regularization techniques might further enhance performance. Surprisingly, neurogenesis, when combined with dropout, weight decay or neural noise, consistently reduced the performance ( Figure 3E). This suggests that the amount of noise injected by neurogenesis may be suboptimal when combined with other regularization techniques. Consistent with this idea, performance improved when the amount of noise injection for each regularization method was reduced. However, even using these reduced parameter values did not enhance performance beyond that of neurogenesis alone, suggesting that there is a ceiling effect beyond which performance cannot be improved further ( Figure 3F).

Targeted neurogenesis in CNNs
In our model, neurogenesis occurs in randomly selected subpopulations of middle layer neurons. However, the integration of new neurons might be non-random in nature. For example, in rodents there is evidence for neurogenesis-dependent refinement of synaptic connections in the hippocampus, with the integration of new neurons leading to the elimination of less active synaptic connections (Yasuda et al., 2011). Here we explored whether implementation of a similarly targeted turnover mechanism during training would further enhance performance of our neurogenic networks. Such an approach has previously been explored using dropout regularization, and, in this case, targeting dropout to neurons that are likely to be less important for the task (i.e., neurons with lowest L1 norm values or total weights) led to significant improvement in test performance (Gomez et al., 2019). Using a similar strategy, here we ranked neurons by their L1 norm values at every turnover event, and reinitialized weights of the bottom 3.2% ranking neurons. As a positive control, we reinitialized the weights of the top 3.2% ranking neurons in a separate experiment ( Figure 4A).
Targeting top-ranked neurons impaired generalization performance below that of the control (no neurogenesis) group. This is expected since these neurons are assumed to be more valuable to the performance of the network. However, targeting the bottomranked neurons did not further enhance performance (i.e, compared to random reinitializations affecting the same size subpopulation) ( Figure 4B). While this finding indicates that targeting turnover neurogenesis to the bottom L1 norm ranking neurons does not further improve generalization, we recognize that there may be alternate ways to determine 'importance' of units in a neurogenic network, and it is possible that targeting these might produce performance improvements.

Neurogenesis improves generalization but increases reliance on individual neurons
The degree to which networks depend on single units vs. more distributed, population codes influences generalization (Morcos et al., 2018). Typically, networks that generalize well on held back, unseen test data depend on distributed population codes, rather than a small subset of units. Moreover, because of their distributed nature, these types of networks tend to be more resilient to random ablation of units (Morcos et al., 2018).
Conversely, networks that generalize poorly tend to depend on a small subset of units, rather than distributed codes, and these networks are typically less resilient to random ablation of units.
We performed a similar analysis here to assess whether neurogenesis improves generalization by reducing the dependence of networks on a small subset of single units.
To evaluate the reliance on single units vs. distributed codes, we sequentially ablated random units following training and compared pre-and post-ablation performance (accuracy was normalized to pre-ablation performance) ( Figure 5A). Surprisingly, we found that neurogenic networks were less resilient. Ablation of a smaller proportion of units was sufficient to decrease performance (as reflected by the leftward shift of the curve) ( Figure 5B). This indicates that neurogenesis improves generalization through a mechanism other than reducing the network's reliance on individual neurons. We also measured the class selectivity for each neuron using the metric described in Morcos et al., (2018) to test whether there were fewer highly class selective neurons. We found that the distribution of class selectivities of neurons was shifted to the left (on average neurons were less selective for class) in the neurogenesis group compared to dropout or control networks ( Figure 5C). To assess whether the reduced resilience to ablation of neurogenic networks translates to increased vulnerability to neurogenesis post-training, we trained networks as before, with and without neurogenesis, and tested performance with and without post-training neurogenesis. One turnover event (replacing 8 neurons) after training/at test-time was used. We found that post-training neurogenesis reduced performance of the neurogenic networks, but not the control networks, further demonstrating a specific sensitivity to perturbation in neurogenic networks ( Figure   5D).

Discussion
As adult-generated neurons integrate into hippocampal circuits they form naive synapses and therefore can be thought of as a form of 'wiring' noise. Deep learning has found that various forms of noise-injection can reduce overfitting on training data and, as a result, enhance generalization in deep neural networks. We therefore hypothesized that neurogenesis-mediated rewiring would similarly have a regularization effect, i.e., prevent memorization of training data and favor a more flexible, generalized memory that can be applied in a broader range of circumstances. We explored this hypothesis computationally, implementing neurogenesis in a hidden layer of a CNN trained on the CIFAR-10 object recognition task. Consistent with our hypothesis, neurogenesis acted as a regularizer, improving generalization on the test (held-out) data. Performance with neurogenesis regularization matched (dropout) or outperformed (weight decay, neural noise) other conventional regularization techniques.
In our model, we implemented neurogenesis as a replacement/turnover mechanism, where input and output synaptic weights associated with small subsets of middle layer neurons were re-initialized through training. We used this approach since neurogenesis in the rodent hippocampus similarly involves replacement of mature, developmentallygenerated neurons with immature, adult-generated neurons, and, therefore, turnover of associated synaptic weights. For instance, in the rat adult dentate gyrus there is significant loss of mature granule cells that were born soon after birth (Ciric et al., 2019). Since there is negligible overall increase in the size of the dentate granule cell layer during adulthood (Rapp & Gallagher, 1996), this implies that this loss of developmentally-generated granule cells in adulthood is balanced by new neuron addition (Cole et al., 2020). Viewed in this way, neurogenesis regularization can be thought of as a form of wiring noise that incrementally alters network connectivity patterns, without impacting overall network size.
In the CNN, we explored whether targeting neurogenesis regularization to less important neurons was beneficial. Using L1 norm values as a metric of "importance" (Gomez et al., 2019), we assessed generalization following three types of neurogenesis regularization: When neurogenesis regularization was applied randomly to all middle layer units versus when neurogenesis regularization was either restricted to the least (i.e., neurons with lowest L1 norm values or weakest weights) or the most (i.e., neurons with highest L1 norm values or strongest weights) important middle layer neurons. As expected, targeting neurons with the highest L1 norm scores decreased generalization scores below control (no neurogenesis) levels, consistent with the idea that neurons with the highest L1 norm values are indeed more important for the task. However, targeting neurogenesis regularization to neurons with the lowest L1 norm scores did not improve generalization beyond networks with non-targeted or random neurogenesis regularization.
In the rodent brain, there is some evidence that the integration of new neurons is a nonrandom process, with the integration of new neurons leading to the pruning of less active synaptic connections (Yasuda et al., 2011). While our current analyses suggest that such a targeted mechanism may not be critical for improving generalization, nonetheless it is possible that using the L1 norm of each unit may not be the most suitable metric of importance. Alternatively, the lottery ticket hypothesis suggests that within these networks there exist sparse sub-networks that, when trained in isolation, can achieve the same final performance accuracy of the entire network but in the same or fewer training epochs (Frankle & Carbin, 2019). Interestingly, the rates at which these lottery-winning sub-network neurons change weights is much higher compared to other neurons. Therefore, one possibility would be to target neurogenesis regularization to neurons that change their weights the least (i.e., those that contribute least to the loss function) during training. Simply looking at the L1 norm, which is not sensitive to the rate of weight changes, would not capture the same "importance" as described in the lottery ticket hypothesis.
A second issue we explored in the CNN was whether neurogenesis regularization impacts how information is organized within the network. Typically, better generalization is associated with more distributed coding (Morcos et al., 2018).
Following dropout regularization, for example, information tends to be coded in a more distributed manner, rather than in single units. One consequence of this organization is that networks tend to be more resilient to random, sequential ablation of units following dropout regularization. In contrast, we found that there was greater reliance on single units following neurogenesis regularization, and networks regularized this way were more vulnerable to random ablations (despite better generalization performance).
One way these findings might be viewed is that while neurogenesis regularization improves generalization it also introduces network vulnerabilities. That is, randomly turning over units pushes the network to rely on neurons that are tuned to single directions (via an unknown mechanism). This may improve generalization performance, but subsequent, random turnover events can also eliminate neurons that are highlytuned to single directions, and hence, have catastrophic consequences. The idea that neurogenesis might introduce network vulnerabilities is interesting since neurogenesis has been linked to forgetting Ryan & Frankland, 2022). For example, post-training increases in neurogenesis induce forgetting of established hippocampus-dependent memories in adult rodents (Akers et al., 2014;Epp et al., 2016;Gao et al., 2018). Whether (or not) forgetting happens may depend both on the levels and/or the timing of neurogenesis.
With respect to levels of neurogenesis, whereas low levels of neurogenesis may promote overfitting (i.e., memorization or no forgetting), high levels of neurogenesis may promote underfitting (i.e., forgetting). In between these extremes, moderate levels of neurogenesis may prevent overfitting and promote generalization (Ko & Frankland, 2021). According to this scenario, it may be the case that neurogenesis levels need to be tightly regulated in order to balance the costs (e.g., overfitting, forgetting) vs. benefits (improved generalization) of rewiring (Richards & Frankland, 2017).
Our analyses suggest that the timing of increases in neurogenesis (with respect to training) also might matter. When neurogenesis occured in concert with training, we found this led to improved performance on subsequent test data. However, when neurogenesis-like turnover occurs at the time of testing we observed a decrease in performance on the unseen test data. This suggests that neurogenic networks need to keep training if they are to retain high performance. Viewed this way, it might be useful to distinguish between the effects of ongoing neurogenesis on active memories (i.e., those that are undergoing training and invulnerable) vs. inactive (i.e., those where training is 'complete' and therefore are vulnerable).
While our model captures replacement/turnover as a core feature of neurogenesis, we nonetheless recognize that it is highly abstracted and differs significantly from biological networks in terms of architecture, connectivity and sparsity. Another feature that was not captured in the current model, but is relevant especially in the context of consolidation, is replay. During sleep, previous event sequences are "replayed" in the hippocampus and this may provide an opportunity to integrate new experiences with prior, relevant experience and promote generalization (Ji & Wilson, 2007;Louie & Wilson, 2001). Indeed, REM sleep, the period of sleep during which replay events typically occur (Louie & Wilson, 2001), is associated with enhanced memory generalization in humans (Lerner et al., 2021). Interestingly, adult-born dentate granule cells that were active during learning are reactivated during subsequent REM (Kumar et al., 2020). This suggests that new neurons may indeed be contributing to consolidation Such a mechanism might underlie generalization through broadening the types of inputs that would drive activation of a given memory.
What are the implications for functional studies of hippocampal neurogenesis? The functional consequences of altering hippocampal neurogenesis (or manipulating the activity of adult-generated granule cells) have largely been studied in rodents (Anacker & Hen, 2017;Cameron & Glover, 2015;Gonçalves et al., 2016). A major focus of these studies has been on the role of hippocampal neurogenesis in pattern separation (Sahay, Wilson, et al., 2011;Santoro, 2013). In a typical experiment, an experimental intervention is introduced to alter levels of hippocampal neurogenesis, and then the ability of mice or rats to make fine spatial or contextual discriminations is assessed. In a touchscreen apparatus, for example, this might involve discriminating between stimuli presented in different spatial locations (Oomen et al., 2013). Rodents with reduced neurogenesis perform poorly under conditions where spatial similarity is high, whereas rodents with elevated neurogenesis exhibit enhanced discrimination (Clelland et al., 2009;Creer et al., 2010).
These results are consistent with the idea that hippocampal neurogenesis regulates a pattern separation-like process in the hippocampus. However, directly comparing these results to our model is difficult given that spatial discrimination is a computationally different task from category discrimination, as we studied here. While studies of category learning are common in monkeys (Ashby & Spiering, 2004), investigations of category learning have been less common in rodents (where neurogenesis levels can be more readily manipulated). Nonetheless, both object recognition and touchscreen-based category learning tasks have been developed for rodents (Broschard et al., 2021;Creighton et al., 2019). The object category recognition tasks take advantage of rodents' innate preference for novelty. During the study phase, mice are allowed to explore two objects from the same category (e.g., two different toy cars). During the test phase, mice are presented with a choice between a third object from the studied category (i.e., another toy car) and a novel object from a new category (e.g., hair clip). Should the mouse exhibit a preference for the object from the unstudied category (i.e., hair clip), then this suggests some within category generalization (i.e., across different types of toy cars) (Creighton et al., 2019). In the touchscreen-based category learning task, mice are trained to discriminate between 2D visual stimuli presented on a touchscreen that can be categorized according to spatial features (e.g., spatial frequency and orientation of gratings). Generalization is assessed when rodents are subsequently tested on novel visual stimuli from the studied categories (Broschard et al., 2021). Based on the findings presented here, we predict that suppression of adult hippocampal neurogenesis will impair generalization in such tests, whereas increasing adult hippocampal neurogenesis will facilitate performance. All of this would be consistent with the effects proposed for neurogenesis in the pattern separation literature.
Neurogenesis has previously been modeled in the context of deep learning in the form of an autoencoder that can perform continual learning. Neurogenesis was found to prevent memory interference of learning new classes on previously learned classes in an image reconstruction task (Draelos et al., 2017). This aligns with other literature where neurogenesis has been shown to improve memory capacity and prevent catastrophic interference (Appleby & Wiskott, 2009;Finnegan & Becker, 2015;Wiskott et al., 2006).
Here, we found that neurogenesis can be applied to CNNs as a novel regularization technique to improve generalization in classification tasks. While the performance of turnover models was superior to neural noise and weight decay, turnover models did not exceed the performance of other regularization methods (namely, dropout). While both methods introduce randomness via the architecture of the network during training, turnover differs from dropout in that turnover consists of the simultaneous permanent loss of learned weights and resetting of weights during the course of training, whereas dropout temporarily silences a random population at each forward pass during training.
Neurogenesis as a regularization technique may benefit further by using a different targeted approach to which neurons are turned over, as discussed above. However, there is also the question as to the mechanism by which neurogenesis improves performance. It is possible that by randomly resetting some neurons during training, these neurons' weights have the opportunity to be arranged in such a way that is more beneficial to training than their previous weights.

Conclusions.
The current analyses provide evidence that 'wiring noise' in the hippocampus, in the form of ongoing neurogenesis, provides a means to regularize memories, thereby preventing memorization and promoting generalization. These ideas are consistent with recent computational and imaging evidence that the hippocampus supports statistical learning/generalized memories, in addition to detailed memories, in humans (Schapiro & Turk-Browne, 2015;Sučević & Schapiro, 2022), and they provide a potential mechanism. Because neurogenesis can be experimentally manipulated in rodents, these predictions can be evaluated using category learning (or related) paradigms in the future.

Computing resources
These experiments were implemented on the high-performance compute clusters at Compute Canada and Vector Institute for AI.

Convolutional neural network
The CNN involves convolutional layers that have filters that convolve to extract features of the image that are not affected by translation. Each convolutional layer is followed by a pooling layer that downsamples the images to improve computational efficiency.
Finally, after multiple sets of convolutional and pooling layers, there are fully connected layers. The input layer was of size 32 ⨉ 32 ⨉ 3, corresponding to the images (with three color channels) from the CIFAR-10 dataset. The CNN was built with three sets of two convolutional layers, each followed by a max-pooling layer. The convolutional layer sets had 16, 32, and 64 filters, respectively, with a 3 ⨉ 3 filter size and stride (steps the filter moves along) of 1. The max-pooling layers pool a 2 ⨉ 2 region and are applied at strides of 1. The convolutional layers are then connected to three fully connected layers, each with 250 units, using a rectified linear activation function. Neurogenesis occurred in the middle, fully connected layer. Data was split into training (40,000 images), validation (10,000 images) and test sets (10,000 images). Test results are derived from networks trained on the full training set. We tuned the network's hyper-parameters by training on the training set (40, 000 images), and testing on the validation set (10, 000 images). The hyper-parameters, i.e., values that regulate learning in the networks, and other information about the networks are listed in tbl. 1.

CIFAR-10
The CIFAR-10 dataset is a collection of images that consists of 50,000 training and 10,000 test images for each of ten classes commonly used to train machine learning and computer vision models. The images are 32 ⨉ 32 color images with the 10 different classes representing airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships and trucks.

Training neurogenic networks
We found that neural networks with neurogenesis can be trained using stochastic gradient descent (Robbins, 2007). During training, neurogenesis was implemented on an ongoing basis, with multiple turnover events occurring. Neurogenesis occured in the second hidden layer (of three). Neurogenesis was implemented as a replacement/turnover mechanism, whereby a randomly chosen subset of neurons in the layer was "turned over" such that their weights were re-initialized to new values. In contrast, the other neurons maintained their learned weights. We initialized the new neurons' weights using the same function used when randomly initializing the network at the start of training, i.e., Kaiming uniform initialization (He et al., 2015). Turnover can occur at a frequency of once every n batch updates. We tuned the hyperparameters using Bayesian Optimization which is an iterative parameter tuning process which builds a probability model for the best parameters to try next (Snoek et al., 2012), whereby we input the turnover frequency (ranging from once every update to once every 12500 updates), and how many neurons to add at each turnover event (ranging from 0 to 250 neurons) as parameters to search for. We found that turning over 8 neurons every 640 updates in the second fully-connected layer resulted in the highest performance on the validation set. We implemented neurogenesis in the output layers of a CNN.

Enhancing excitability
To evaluate whether manipulating "excitability" of new neurons might impact regularization, in a separate experiment we incorporated an excitability component into our neural networks in each forward pass during training. We multiplied the activations of the neurogenesis layer by an excitability array. The activations corresponding to the new neurons were set to a factor (1.3; determined by hyperparameter tuning), and all other activations were set to 1. As a result, only the activations corresponding to the turned-over neurons were enhanced, and the remainder of the neurons' activations were unchanged. Whenever a new set of neurons were turned over, the previous set of new neurons became "mature" and were no longer "excited" in the forward pass.

Other Regularization Methods
To compare the performance of neurogenesis to other regularization methods, we also used dropout, weight decay and neural noise in the CNNs. Dropout involves the stochastic silencing of a subset of units during each forward pass (Srivastava et al., 2014). We used a dropout rate of 0.2 in CNNs, with an additional 7 epochs required to reach the end of training, compared to non-dropout networks. Weight decay consists of adding a small penalty to the loss function that penalizes large weights, thus resulting in an overall decay of larger weights (Krogh & Hertz, 1992). We used a weight decay value of 0.00001 for our CNN. Neural noise involves the addition of Gaussian noise to all the activations during each forward pass (Bishop, 1995). We defined our noise using a mean of -0.2, and a standard deviation of 0.5 on a log normal distribution for our CNN.

Ablation experiments
To measure the importance of individual neurons in a network, we tested how much the network's performance degrades as progressively more neurons are removed from the network (based on Morcos et al. (2018)). To remove a neuron, we set that neuron's activity to a fixed value of 0, effectively ablating the unit. We progressively ablated neurons in proportional steps of 5% of the neurons in the neurogenesis layer, testing the accuracy of the training data at each step to generate ablation curves. We repeated each ablation five times and randomized the order of neurons ablated each time. Ablation curves plot the degradation in accuracy as more neurons are ablated. Networks that rely more heavily on individual units will drop their accuracy more quickly as units are ablated. Networks that are less sensitive to ablations have been shown to correlate with better generalization (Morcos et al., 2018).

Class selectivity
Class selectivity was calculated using the method described in Morcos et al., 2018. The class-conditional mean activity was calculated from the test set and the class selectivity index was measured as where ��� is the highest mean class activity and −��� is the mean class activity across all other classes.

Targeted neurogenesis
To test whether a targeted approach to removing new neurons might improve the performance of neurogenic networks, we implemented a targeted turnover of new

Statistical analyses and plotting
All statistics were performed using the scipy.stats module in Python 3.6 (Van Rossum & Drake, 2009; Virtanen et al., 2020). Error bars on graphs represent the standard error of the mean across different initializations of the model, where each experiment is repeated 20 times unless otherwise stated. Comparisons were made using unpaired, two-tailed t-tests, or analysis of variance (ANOVA) followed by Tukey's HSD post-hoc tests where appropriate. Significance indicated by an asterisk (*) for p-values less than 0.01 unless otherwise stated. Graphs were generated using the matplotlib, and seaborn packages in Python 3.6 (Hunter, 2007;Waskom et al., 2017), and figures were compiled using Inkscape (Inkscape Project, 2020).

Author contributions
LMT contributed to conceptualization, data curation, formal analyses, investigation, methodology, software, validation, visualization, and writing (original draft preparation, review and editing). AS contributed to conceptualization and writing (review and editing). LL contributed to formal analysis and software. SAJ contributed to funding acquisition and writing (review and editing). BAR contributed to conceptualization, funding acquisition and writing (review and editing). PWF contributed to conceptualization, supervision, funding acquisition and writing (original draft preparation, review and editing).