Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

The impact of different sources of heterogeneity on loss of accuracy from genomic prediction models

Yuqing Zhang, Christoph Bernau, Giovanni Parmigiani, Levi Waldron
doi: https://doi.org/10.1101/374355
Yuqing Zhang
Graduate Program in Bioinformatics, Boston University, Boston, U.S.A
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Christoph Bernau
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Giovanni Parmigiani
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Levi Waldron
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Supplementary material
  • Preview PDF
Loading

SUMMARY

Cross-study validation (CSV) of prediction models is an alternative to traditional cross-validation (CV) in domains where multiple comparable datasets are available. Although many studies have noted potential sources of heterogeneity in genomic studies, to our knowledge none have system atically investigated their intertwined impacts on prediction accuracy across studies. We employ a hybrid parametric/non-parametric bootstrap method to realistically simulate publicly available compendia of microarray, RNA-seq, and whole metagenome shotgun (WMS) microbiome studies of health outcomes. Three types of heterogeneity between studies are manipulated and studied: imbalances in the prevalence of clinical and pathological covariates, 2) differences in gene covariance that could be caused by batch, platform, or tumor purity effects, and 3) differences in the “true” model that associates gene expression and clinical factors to outcome. We assess model accuracy while altering these factors. Lower accuracy is seen in CSV than in CV. Surprisingly, heterogeneity in known clinical covariates and differences in gene covariance structure have very limited contributions in the loss of accuracy when validating in new studies. However, forcing identical generative models greatly reduces the within/across study difference. These results, observed consistently for multiple disease outcomes and omics platforms, suggest that the most easily identifiable sources of study heterogeneity are not necessarily the primary ones that undermine the ability to accurately replicate the accuracy of omics prediction models in new studies. Unidentified heterogeneity, such as could arise from unmeasured confounding, may be more important.

Copyright 
The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-ND 4.0 International license.
Back to top
PreviousNext
Posted July 23, 2018.
Download PDF

Supplementary Material

Email

Thank you for your interest in spreading the word about bioRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
The impact of different sources of heterogeneity on loss of accuracy from genomic prediction models
(Your Name) has forwarded a page to you from bioRxiv
(Your Name) thought you would like to see this page from the bioRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
The impact of different sources of heterogeneity on loss of accuracy from genomic prediction models
Yuqing Zhang, Christoph Bernau, Giovanni Parmigiani, Levi Waldron
bioRxiv 374355; doi: https://doi.org/10.1101/374355
Digg logo Reddit logo Twitter logo Facebook logo Google logo LinkedIn logo Mendeley logo
Citation Tools
The impact of different sources of heterogeneity on loss of accuracy from genomic prediction models
Yuqing Zhang, Christoph Bernau, Giovanni Parmigiani, Levi Waldron
bioRxiv 374355; doi: https://doi.org/10.1101/374355

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Bioinformatics
Subject Areas
All Articles
  • Animal Behavior and Cognition (3497)
  • Biochemistry (7341)
  • Bioengineering (5317)
  • Bioinformatics (20248)
  • Biophysics (9999)
  • Cancer Biology (7734)
  • Cell Biology (11291)
  • Clinical Trials (138)
  • Developmental Biology (6431)
  • Ecology (9943)
  • Epidemiology (2065)
  • Evolutionary Biology (13311)
  • Genetics (9358)
  • Genomics (12575)
  • Immunology (7696)
  • Microbiology (18998)
  • Molecular Biology (7432)
  • Neuroscience (40971)
  • Paleontology (300)
  • Pathology (1228)
  • Pharmacology and Toxicology (2133)
  • Physiology (3154)
  • Plant Biology (6855)
  • Scientific Communication and Education (1272)
  • Synthetic Biology (1895)
  • Systems Biology (5309)
  • Zoology (1087)