RT Journal Article
SR Electronic
T1 The EFFECT benchmark suite: measuring cancer sensitivity prediction performance - without the bias
JF bioRxiv
FD Cold Spring Harbor Laboratory
SP 2023.10.02.560281
DO 10.1101/2023.10.02.560281
A1 Szalai, Bence
A1 Gáspár, Imre
A1 Kaszás, Valér
A1 Mérő, László
A1 Sztilkovics, Milán
A1 Szalay, Kristóf Z.
YR 2023
UL http://biorxiv.org/content/early/2023/10/30/2023.10.02.560281.abstract
AB Creating computational biology models applicable to industry is much more difficult than it appears. There is a major gap between a model that looks good on paper and a model that performs well in the drug discovery process. We are trying to shrink this gap by introducing the Evaluation Framework For predicting Efficiency of Cancer Treatment (EFFECT) benchmark suite based on the DepMap and GDSC data sets to facilitate the creation of well-applicable machine learning models capable of predicting gene essentiality and/or drug sensitivity on in vitro cancer cell lines.We show that standard evaluation metrics like Pearson correlation are misleading due to inherent biases in the data. Thus, to assess the performance of models properly, we propose the use of cell line/perturbation exclusive data splits, perturbation-wise evaluation, and the application of our Bias Detector framework, which can identify model predictions not explicable by data bias alone.Testing the EFFECT suite on a few popular machine learning (ML) models showed that while library-standard non-linear models have measurable performance in splits representing precision medicine and target identification tasks, the actual corrected correlations are rather low, showing that even simple knock-out (KO)/drug sensitivity prediction is a yet unsolved task.For this reason, we aim our proposed framework to be a unified test and evaluation pipeline for ML models predicting cancer sensitivity data, facilitating unbiased benchmarking to support teams to improve on the state of the art.Competing Interest StatementAll authors are full-time employees of Turbine Ltd., KSz is a founder as well.