RT Journal Article SR Electronic T1 The geometry of hidden representations of protein language models JF bioRxiv FD Cold Spring Harbor Laboratory SP 2022.10.24.513504 DO 10.1101/2022.10.24.513504 A1 Valeriani, Lucrezia A1 Cuturello, Francesca A1 Ansuini, Alessio A1 Cazzaniga, Alberto YR 2022 UL http://biorxiv.org/content/early/2022/10/26/2022.10.24.513504.abstract AB Protein language models (pLMs) transform their input into a sequence of hidden representations whose geometric behavior changes across layers. Looking at fundamental geometric properties such as the intrinsic dimension and the neighbor composition of these representations, we observe that these changes highlight a pattern characterized by three distinct phases. This phenomenon emerges across many models trained on diverse datasets, thus revealing a general computational strategy learned by pLMs to reconstruct missing parts of the data. These analyses show the existence of low-dimensional maps that encode evolutionary and biological properties such as remote homology and structural information. Our geometric approach sets the foundations for future systematic attempts to understand the space of protein sequences with representation learning techniques.Competing Interest StatementThe authors have declared no competing interest.