RT Journal Article SR Electronic T1 LC-PLM: Long-context Protein Language Model JF bioRxiv FD Cold Spring Harbor Laboratory SP 2024.10.29.620988 DO 10.1101/2024.10.29.620988 A1 Wang, Yingheng A1 Wang, Zichen A1 Sadeh, Gil A1 Zancato, Luca A1 Achille, Alessandro A1 Karypis, George A1 Rangwala, Huzefa YR 2024 UL http://biorxiv.org/content/early/2024/10/30/2024.10.29.620988.abstract AB Self-supervised training of language models (LMs) has seen great success for protein sequences in learning meaningful representations and for generative drug design. Most protein LMs are based on the Transformer architecture trained on individual proteins with short context lengths. Such protein LMs cannot extrapolate to longer proteins and protein complexes well. They also fail to account for the underlying biological mechanisms carried out by biomolecular interactions and dynamics i.e., proteins often interact with other proteins, molecules, and pathways in complex biological systems. In this work, we propose LC-PLM based on an alternative protein LM architecture, BiMamba-S, built off selective structured state-space models, to learn high-quality universal protein representations at the amino acid token level using masked language modeling. We also introduce its graph-contextual variant, LC-PLM-G, which contextualizes protein-protein interaction (PPI) graphs for a second stage of training. LC-PLM demonstrates favorable neural scaling laws, better length extrapolation capability, and a 7% to 34% improvement on protein downstream tasks than Transformer-based ESM-2. LC-PLM-G further trained within the context of PPI graphs shows promising results on protein structure and function prediction tasks. Our study demonstrates the benefit of increasing the context size with computationally efficient LM architecture (e.g. structured state space models) in learning universal protein representations and incorporating molecular interaction context contained in biological graphs.Competing Interest StatementThe authors have declared no competing interest.