PT  - JOURNAL ARTICLE
AU  - Adam C. Mater
AU  - Mahakaran Sandhu
AU  - Colin Jackson
TI  - The NK Landscape as a Versatile Benchmark for Machine Learning Driven Protein Engineering
AID  - 10.1101/2020.09.30.319780
DP  - 2020 Jan 01
TA  - bioRxiv
PG  - 2020.09.30.319780
4099  - http://biorxiv.org/content/early/2020/10/06/2020.09.30.319780.short
4100  - http://biorxiv.org/content/early/2020/10/06/2020.09.30.319780.full
AB  - Machine learning (ML) has the potential to revolutionize protein engineering. However, the field currently lacks standardized and rigorous evaluation benchmarks for sequence-fitness prediction, which makes accurate evaluation of the performance of different architectures difficult. Here we propose a unifying framework for ML-driven sequence-fitness prediction. Using simulated (the NK model) and empirical sequence landscapes, we define four key performance metrics: interpolation within the training domain, extrapolation outside the training domain, robustness to sparse training data, and ability to cope with epistasis/ruggedness. We show that architectural differences between algorithms consistently affect performance against these metrics across both experimental and theoretical landscapes. Moreover, landscape ruggedness is revealed to be the greatest determinant of the accuracy of sequence-fitness prediction. We hope that this benchmarking method and the code that accompanies it will enable robust evaluation and comparison of novel architectures in this emerging field and assist in the adoption of ML for protein engineering.Competing Interest StatementThe authors have declared no competing interest.