PT  - JOURNAL ARTICLE
AU  - Wang, Yingheng
AU  - Wang, Zichen
AU  - Sadeh, Gil
AU  - Zancato, Luca
AU  - Achille, Alessandro
AU  - Karypis, George
AU  - Rangwala, Huzefa
TI  - LC-PLM: Long-context Protein Language Model
AID  - 10.1101/2024.10.29.620988
DP  - 2024 Jan 01
TA  - bioRxiv
PG  - 2024.10.29.620988
4099  - http://biorxiv.org/content/early/2024/10/30/2024.10.29.620988.short
4100  - http://biorxiv.org/content/early/2024/10/30/2024.10.29.620988.full
AB  - Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design. Most protein LMs are based on the Transformer architecture trained on individual proteins with short context lengths. Such protein LMs cannot extrapolate to longer proteins and protein complexes well. They also fail to account for the underlying biological mechanisms carried out by biomolecular interactions and dynamics i.e., proteins often interact with other proteins, molecules, and pathways in complex biological systems. In this work, we propose LC-PLM based on an alternative protein LM architecture, BiMamba-S, built off selective structured state-space models, to learn high-quality universal protein representations at the amino acid token level using masked language modeling. We also introduce its graph-contextual variant, LC-PLM-G, which contextualizes protein-protein interaction (PPI) graphs for a second stage of training. LC-PLM demonstrates favorable neural scaling laws, better length extrapolation capability, and a 7% to 34% improvement on protein downstream tasks than Transformer-based ESM-2. LC-PLM-G further trained within the context of PPI graphs shows promising results on protein structure and function prediction tasks. Our study demonstrates the benefit of increasing the context size with computationally efficient LM architecture (e.g. structured state space models) in learning universal protein representations and incorporating molecular interaction context contained in biological graphs.Competing Interest StatementThe authors have declared no competing interest.