Abstract
Adapting large language models (LLMs) to protein sequences spawned the development of powerful protein language models (pLMs). Concurrently, AlphaFold2 broke through in protein structure prediction. Now we can systematically and comprehensively explore the dual nature of proteins that act and exist as three-dimensional (3D) machines and evolve as linear strings of one-dimensional (1D) sequences. Here, we leverage pLMs to simultaneously model both modalities by combining 1D sequences with 3D structure in a single model. We encode protein structures as token sequences using the 3Di-alphabet introduced by the 3D-alignment method Foldseek. This new foundation pLM extracts the features and patterns of the resulting “structure-sequence” representation. Toward this end, we built a non-redundant dataset from AlphaFoldDB and fine-tuned an existing pLM (ProtT5) to translate between 3Di and amino acid sequences. As a proof-of-concept for our novel approach, dubbed Protein structure-sequence T5 (ProstT5), we showed improved performance for subsequent prediction tasks, and for “inverse folding”, namely the generation of novel protein sequences adopting a given structural scaffold (“fold”). Our work showcased the potential of pLMs to tap into the information-rich protein structure revolution fueled by AlphaFold2. ProstT5 paves the way to develop new tools integrating the vast resource of 3D predictions, and opens new research avenues in the post-AlphaFold2 era. Our model is freely available for all at https://github.com/mheinzinger/ProstT5.
Competing Interest Statement
The authors have declared no competing interest.
Footnotes
- Expanded results: faster 3Di inference from AA sequences thanks to CNN trained directly on ProstT5-encoder (avoids sequential generation via decoder) - Expanded analysis: speed comparison of CNN-based 3Di prediction to colabFold and full ProstT5 (w. Encoder and Decoder) - Improved illustrations (Fig. 1 & Fig. S1) - Text refinement & slight change of focus (more weight on remote homology detection)
Abbreviations & Glossary
- 1D
- one-dimensional (string such as secondary structure)
- 3D
- three-dimensional (coordinates)
- 3Di
- 1D-strings representing protein 3D structure (taken from Foldseek1)
- AA
- amino acid
- AFDB
- AlphaFold Protein Structure Database
- CATH
- hierarchical classification of protein 3D structures in Class, Architecture, Topology and Homologous superfamily
- CNN
- convolutional neural network
- EAT
- Embedding-based Annotation Transfer; embeddings fixed-size vectors derived from pre-trained pLMs
- LLM
- large language model
- pLM
- protein Language Model
- SOTA
- state-of-the-art.