RT Journal Article
SR Electronic
T1 The geometry of hidden representations of protein language models
JF bioRxiv
FD Cold Spring Harbor Laboratory
SP 2022.10.24.513504
DO 10.1101/2022.10.24.513504
A1 Lucrezia Valeriani
A1 Francesca Cuturello
A1 Alessio Ansuini
A1 Alberto Cazzaniga
YR 2022
UL http://biorxiv.org/content/early/2022/10/26/2022.10.24.513504.abstract
AB Protein language models (pLMs) transform their input into a sequence of hidden representations whose geometric behavior changes across layers. Looking at fundamental geometric properties such as the intrinsic dimension and the neighbor composition of these representations, we observe that these changes highlight a pattern characterized by three distinct phases. This phenomenon emerges across many models trained on diverse datasets, thus revealing a general computational strategy learned by pLMs to reconstruct missing parts of the data. These analyses show the existence of low-dimensional maps that encode evolutionary and biological properties such as remote homology and structural information. Our geometric approach sets the foundations for future systematic attempts to understand the space of protein sequences with representation learning techniques.Competing Interest StatementThe authors have declared no competing interest.