Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

Best Practices for Benchmarking Germline Small Variant Calls in Human Genomes

Peter Krusche, Len Trigg, Paul C. Boutros, Christopher E. Mason, Francisco M. De La Vega, Benjamin L. Moore, Mar Gonzalez-Porta, Michael A. Eberle, Zivana Tezak, Samir Lababidi, Rebecca Truty, George Asimenos, Birgit Funke, Mark Fleharty, Marc Salit, Justin M Zook
doi: https://doi.org/10.1101/270157
Peter Krusche
1Illumina Cambridge Ltd, Chesterford Research Park, Little Chesterford, Saffron Walden, Essex, CB10 1XL, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Len Trigg
2Real Time Genomics, Level 1, South Bloc, 19 Knox St. Hamilton. New Zealand
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Paul C. Boutros
3Ontario Institute for Cancer Research, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Christopher E. Mason
4Weill Cornell Medicine, New York, NY, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Francisco M. De La Vega
5Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Benjamin L. Moore
1Illumina Cambridge Ltd, Chesterford Research Park, Little Chesterford, Saffron Walden, Essex, CB10 1XL, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Mar Gonzalez-Porta
1Illumina Cambridge Ltd, Chesterford Research Park, Little Chesterford, Saffron Walden, Essex, CB10 1XL, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michael A. Eberle
6Illumina Inc., 5200 Illumina Way, San Diego, CA 92122
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Zivana Tezak
7Center for Devices and Radiological Health, FDA, Silver Spring, MD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Samir Lababidi
8Office of Health Informatics, Office of the Commissioner, FDA, Silver Spring, MD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Rebecca Truty
9Invitae, 1400 16th St, San Francisco, CA 94103
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
George Asimenos
10DNAnexus, 730 Market St Suite 2100, San Francisco, CA 94103
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Birgit Funke
11Veritas Genetics, 99 Conifer Hill Dr, Danvers, MA 01923
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Mark Fleharty
12Broad Institute, 415 Main Street, Cambridge, MA 02142, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Marc Salit
13Material Measurement Laboratory, Joint Initiative for Metrology in Biology, National Institute of Standards and Technology, Stanford, CA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Justin M Zook
14Material Measurement Laboratory, National Institute of Standards and Technology, 100 Bureau Dr, MS8301, Gaithersburg, MD 20899
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Supplementary material
  • Preview PDF
Loading

Abstract

Assessing accuracy of NGS variant calling is immensely facilitated by a robust benchmarking strategy and tools to carry it out in a standard way. Benchmarking variant calls requires careful attention to definitions of performance metrics, sophisticated comparison approaches, and stratification by variant type and genome context. The Global Alliance for Genomics and Health (GA4GH) Benchmarking Team has developed standardized performance metrics and tools for benchmarking germline small variant calls. This Team includes representatives from sequencing technology developers, government agencies, academic bioinformatics researchers, clinical laboratories, and commercial technology and bioinformatics developers for whom benchmarking variant calls is essential to their work. Benchmarking variant calls is a challenging problem for many reasons:

  • Evaluating variant calls requires complex matching algorithms and standardized counting, because the same variant may be represented differently in truth and query callsets.

  • Defining and interpreting resulting metrics such as precision (aka positive predictive value = TP/(TP+FP)) and recall (aka sensitivity = TP/(TP+FN)) requires standardization to draw robust conclusions about comparative performance for different variant calling methods.

  • Performance of NGS methods can vary depending on variant types and genome context; and as a result understanding performance requires meaningful stratification.

  • High-confidence variant calls and regions that can be used as “truth” to accurately identify false positives and negatives are difficult to define, and reliable calls for the most challenging regions and variants remain out of reach.

We have made significant progress on standardizing comparison methods, metric definitions and reporting, as well as developing and using truth sets. Our methods are publicly available on GitHub (https://github.com/ga4gh/benchmarking-tools) and in a web-based app on precisionFDA, which allow users to compare their variant calls against truth sets and to obtain a standardized report on their variant calling performance. Our methods have been piloted in the precisionFDA variant calling challenges to identify the best-in-class variant calling methods within high-confidence regions. Finally, we recommend a set of best practices for using our tools and critically evaluating the results.

Copyright 
The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY 4.0 International license.
Back to top
PreviousNext
Posted February 23, 2018.
Download PDF

Supplementary Material

Email

Thank you for your interest in spreading the word about bioRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Best Practices for Benchmarking Germline Small Variant Calls in Human Genomes
(Your Name) has forwarded a page to you from bioRxiv
(Your Name) thought you would like to see this page from the bioRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Best Practices for Benchmarking Germline Small Variant Calls in Human Genomes
Peter Krusche, Len Trigg, Paul C. Boutros, Christopher E. Mason, Francisco M. De La Vega, Benjamin L. Moore, Mar Gonzalez-Porta, Michael A. Eberle, Zivana Tezak, Samir Lababidi, Rebecca Truty, George Asimenos, Birgit Funke, Mark Fleharty, Marc Salit, Justin M Zook
bioRxiv 270157; doi: https://doi.org/10.1101/270157
Digg logo Reddit logo Twitter logo Facebook logo Google logo LinkedIn logo Mendeley logo
Citation Tools
Best Practices for Benchmarking Germline Small Variant Calls in Human Genomes
Peter Krusche, Len Trigg, Paul C. Boutros, Christopher E. Mason, Francisco M. De La Vega, Benjamin L. Moore, Mar Gonzalez-Porta, Michael A. Eberle, Zivana Tezak, Samir Lababidi, Rebecca Truty, George Asimenos, Birgit Funke, Mark Fleharty, Marc Salit, Justin M Zook
bioRxiv 270157; doi: https://doi.org/10.1101/270157

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Genomics
Subject Areas
All Articles
  • Animal Behavior and Cognition (4112)
  • Biochemistry (8814)
  • Bioengineering (6518)
  • Bioinformatics (23459)
  • Biophysics (11789)
  • Cancer Biology (9206)
  • Cell Biology (13321)
  • Clinical Trials (138)
  • Developmental Biology (7435)
  • Ecology (11409)
  • Epidemiology (2066)
  • Evolutionary Biology (15147)
  • Genetics (10436)
  • Genomics (14042)
  • Immunology (9170)
  • Microbiology (22152)
  • Molecular Biology (8811)
  • Neuroscience (47564)
  • Paleontology (350)
  • Pathology (1428)
  • Pharmacology and Toxicology (2491)
  • Physiology (3730)
  • Plant Biology (8079)
  • Scientific Communication and Education (1437)
  • Synthetic Biology (2220)
  • Systems Biology (6037)
  • Zoology (1253)