RT Journal Article
SR Electronic
T1 SEPARABLE FULLY CONNECTED LAYERS IMPROVE DEEP LEARNING MODELS FOR GENOMICS
JF bioRxiv
FD Cold Spring Harbor Laboratory
SP 146431
DO 10.1101/146431
A1 Amr Mohamed Alexandari
A1 Avanti Shrikumar
A1 Anshul Kundaje
YR 2017
UL http://biorxiv.org/content/early/2017/06/10/146431.abstract
AB Convolutional neural networks are rapidly gaining popularity in regulatory genomics. Typically, these networks have a stack of convolutional and pooling layers, followed by one or more fully connected layers. In genomics, the same positional patterns are often present across multiple convolutional channels. Therefore, in current state-of-the-art networks, there exists significant redundancy in the representations learned by standard fully connected layers. We present a new separable fully connected layer that learns a weights tensor that is the outer product of positional weights and cross-channel weights, thereby allowing the same positional patterns to be applied across multiple convolutional channels. Decomposing positional and cross-channel weights further enables us to readily impose biologically-inspired constraints on positional weights, such as symmetry. We also propose a novel regularizer that penalizes curvature in the positional weights. Using experiments on simulated and in vivo datasets, we show that networks that incorporate our separable fully connected layer outperform conventional models with analogous architectures and the same number of parameters. Additionally, our networks are more robust to hyperparameter tuning, have more informative gradients, and produce importance scores that are more consistent with known biology than conventional deep neural networks.Availability Our implementation is available at: https://github.com/kundajelab/keras/tree/keras_1A gist illustrating model setup is at: goo.gl/gYooaa