PT - JOURNAL ARTICLE
AU - Quax, Silvan C.
AU - Dâ€™Asaro, Michele
AU - van Gerven, Marcel A. J.
TI - Adaptive time scales in recurrent neural networks
AID - 10.1101/800540
DP - 2019 Jan 01
TA - bioRxiv
PG - 800540
4099 - http://biorxiv.org/content/early/2019/10/10/800540.short
4100 - http://biorxiv.org/content/early/2019/10/10/800540.full
AB - Recurrent neural network models have become widely used in computational neuroscience to model the dynamics of neural populations as well as in machine learning applications to model data with temporal dependencies. The different variants of RNNs commonly used in these scientific fields can be derived as discrete time approximations of the instantaneous firing rate of a population of neurons. The time constants of the neuronal process are generally ignored in these approximations, while learning these time constants could possibly inform us about the time scales underlying temporal processes and enhance the expressive capacity of the network. To investigate the potential of adaptive time constants, we compare the standard Elman approximation to a more lenient one that still accounts for the time scales at which processes unfold. We show that such a model with adaptive time scales performs better on predicting temporal data, increasing the memory capacity of recurrent neural networks, and allows recovery of the time scales at which the underlying processes unfold.