Abstract
GO classifiers and other methods for automatic annotations of novel sequences play an important role in modern biosciences. It is thus important to assess the quality of different GO classifiers. Evaluation of GO classifiers depends heavily on the used evaluation metrics. Still, there has been little research on the effect of different metrics on the produced method ranking. Indeed most evaluation metrics are simply borrowed from machine learning without any testing for their applicability to GO classification.
We propose a novel simple comparison of metrics, called Artificial Dilution Series. We start by selecting a set of annotations that are known a priori to be correct. From this set we create multiple copies and introduce different amount of errors to each copy. This creates a “series” of annotation sets with the percentage of original correct annotations (“signal”) decreasing from one end of the series to the other. Next we test metrics to see which of them are good at separating annotation sets at different signal levels. In addition, we test metrics with various false positive annotation sets, and show where they rank in the generated signal range.
We compared a large set of evaluation metrics with ADS, revealing drastic differences between them. Especially we show how some metrics consider false positive datasets as good as 100 % correct data sets and how some metrics perform poorly at separating the different error levels. This work A) shows that evaluation metrics should be tested for their performance; B) presents a software that can be used to test different metrics on real-life datasets; C) gives guide lines on what evaluation metrics perform well with Gene Ontology structure; D) proposes improved versions for some well-known evaluation metrics. The presented methods are also applicable to other areas of science where evaluation of prediction results is non-trivial.
Author Summary Comparison of predictive methods is one of the central tasks in science and bioinformatics is no exception. Currently predictive methods are increasingly needed as biosciences are producing novel sequences at an ever higher rate. These sequences require Automated Function Prediction (AFP) as manual curation is often impossible. Unfortunately, selecting AFP methods is a confusing task as current AFP publications show a mixed set of functions for method comparison called Evaluation Metrics (metrics for short). Furthermore, many existing popular metrics can generate misleading or unreasonable results in AFP comparison. We argue that the usage of badly performing metrics in AFP comparison is caused by the lack of methods that can be used to benchmark the metrics. We propose a testing method, called Artificial Dilution Series (ADS). It can be used to test any group of metrics on selected real life test dataset. ADS uses selected dataset to create a large set of artificial AFP results, where each AFP result has a controlled amount of errors. We use ADS to compare how different metrics are able to separate generated error proportions. Our results show drastic differences between different metrics.
