RT Journal Article
SR Electronic
T1 Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
JF bioRxiv
FD Cold Spring Harbor Laboratory
SP 622803
DO 10.1101/622803
A1 Rives, Alexander
A1 Meier, Joshua
A1 Sercu, Tom
A1 Goyal, Siddharth
A1 Lin, Zeming
A1 Liu, Jason
A1 Guo, Demi
A1 Ott, Myle
A1 Zitnick, C. Lawrence
A1 Ma, Jerry
A1 Fergus, Rob
YR 2020
UL http://biorxiv.org/content/early/2020/12/15/622803.abstract
AB In the field of artificial intelligence, a combination of scale in data and model capacity enabled by un-supervised learning has led to major advances in representation learning and statistical generation. In the life sciences, the anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To this end we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250 million protein sequences spanning evolutionary diversity. The resulting model contains information about biological properties in its representations. The representations are learned from sequence data alone. The learned representation space has a multi-scale organization reflecting structure from the level of biochemical properties of amino acids to remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and can be identified by linear projections. Representation learning produces features that generalize across a range of applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure, and improving state-of-the-art features for long-range contact prediction.Competing Interest StatementThe authors have declared no competing interest.