RT Journal Article SR Electronic T1 Convolutions are competitive with transformers for protein sequence pretraining JF bioRxiv FD Cold Spring Harbor Laboratory SP 2022.05.19.492714 DO 10.1101/2022.05.19.492714 A1 Kevin K. Yang A1 Alex X. Lu A1 Nicolo Fusi YR 2022 UL http://biorxiv.org/content/early/2022/05/20/2022.05.19.492714.abstract AB Pretrained protein sequence language models largely rely on the transformer architecture. However, transformer run-time and memory requirements scale quadrat-ically with sequence length. We investigate the potential of a convolution-based architecture for protein sequence masked language model pretraining and subsequent finetuning. CNNs are competitive on the pretraining task with transformers across several orders of magnitude in parameter size while scaling linearly with sequence length. More importantly, CNNs are competitive with and occasionally superior to transformers across an extensive set of downstream evaluations, including structure prediction, zero-shot mutation effect prediction, and out-of-domain generalization. We also demonstrate strong performance on sequences longer than the positional embeddings allowed in the current state-of-the-art transformer protein masked language models. Finally, we close with a call to disentangle the effects of pretraining task and model architecture when studying pretrained protein sequence models.Competing Interest StatementThe authors have declared no competing interest.