Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

Deep learning from multiple experts improves identification of amyloid neuropathologies

View ORCID ProfileDaniel R. Wong, View ORCID ProfileZiqi Tang, View ORCID ProfileNicholas C. Mew, Sakshi Das, Justin Athey, View ORCID ProfileKirsty E. McAleese, Julia K. Kofler, Margaret E. Flanagan, Ewa Borys, View ORCID ProfileCharles L. White III, View ORCID ProfileAtul J. Butte, View ORCID ProfileBrittany N. Dugger, View ORCID ProfileMichael J. Keiser
doi: https://doi.org/10.1101/2021.03.12.435050
Daniel R. Wong
1Bakar Computational Health Sciences Institute, University of California, San Francisco, CA, 94158, USA
2Institute for Neurodegenerative Diseases, University of California, San Francisco, CA, 94158, USA
3Department of Pharmaceutical Chemistry, University of California, San Francisco, CA, 94158, USA
4Department of Bioengineering and Therapeutic Sciences, University of California, San Francisco, CA, 94158, USA
5Department of Pediatrics, University of California, San Francisco, CA, 94158, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Daniel R. Wong
Ziqi Tang
2Institute for Neurodegenerative Diseases, University of California, San Francisco, CA, 94158, USA
3Department of Pharmaceutical Chemistry, University of California, San Francisco, CA, 94158, USA
4Department of Bioengineering and Therapeutic Sciences, University of California, San Francisco, CA, 94158, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Ziqi Tang
Nicholas C. Mew
2Institute for Neurodegenerative Diseases, University of California, San Francisco, CA, 94158, USA
3Department of Pharmaceutical Chemistry, University of California, San Francisco, CA, 94158, USA
4Department of Bioengineering and Therapeutic Sciences, University of California, San Francisco, CA, 94158, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Nicholas C. Mew
Sakshi Das
6Department of Pathology and Laboratory Medicine, School of Medicine, University of California, Davis, Sacramento, CA 95817, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Justin Athey
6Department of Pathology and Laboratory Medicine, School of Medicine, University of California, Davis, Sacramento, CA 95817, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Kirsty E. McAleese
7Translation and Clinical Research Institute, Newcastle University, Newcastle, United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Kirsty E. McAleese
Julia K. Kofler
8Department of Pathology, University of Pittsburgh Medical Center, Pittsburgh, PA, 15260, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Margaret E. Flanagan
9Department of Pathology, Northwestern University, Evanston, IL, 60208, USA
10Mesulam Center for Cognitive Neurology and Alzheimer’s Disease, Northwestern Medicine, Chicago, IL, 60611, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ewa Borys
11Department of Pathology, Loyola University Medical Center, Maywood, IL, 60153, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Charles L. White III
12Department of Pathology, University of Texas Southwestern Medical Center, Dallas, TX, 75390, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Charles L. White III
Atul J. Butte
1Bakar Computational Health Sciences Institute, University of California, San Francisco, CA, 94158, USA
5Department of Pediatrics, University of California, San Francisco, CA, 94158, USA
13Center for Data-Driven Insights and Innovation, University of California, Office of the President, Oakland, CA, 94607, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Atul J. Butte
Brittany N. Dugger
6Department of Pathology and Laboratory Medicine, School of Medicine, University of California, Davis, Sacramento, CA 95817, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Brittany N. Dugger
  • For correspondence: bndugger@ucdavis.edu keiser@keiserlab.org
Michael J. Keiser
1Bakar Computational Health Sciences Institute, University of California, San Francisco, CA, 94158, USA
2Institute for Neurodegenerative Diseases, University of California, San Francisco, CA, 94158, USA
3Department of Pharmaceutical Chemistry, University of California, San Francisco, CA, 94158, USA
4Department of Bioengineering and Therapeutic Sciences, University of California, San Francisco, CA, 94158, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Michael J. Keiser
  • For correspondence: bndugger@ucdavis.edu keiser@keiserlab.org
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Supplementary material
  • Data/Code
  • Preview PDF
Loading

Abstract

Pathologists can label pathologies differently, making it challenging to yield consistent assessments in the absence of one ground truth. To address this problem, we present a DL approach that draws on a cohort of experts, weighs each contribution, and is robust to noisy labels. We collected 100,495 annotations on 20,099 candidate amyloid beta neuropathologies (cerebral amyloid angiopathy (CAA), and cored and diffuse plaques) from three institutions, independently annotated by five experts. DL methods trained on a consensus-of-two strategy yielded 12.6-26% improvements by area under the precision recall curve (AUPRC) when compared to those that learned individualized annotations. This strategy surpassed individual-expert models, even when unfairly assessed on benchmarks favoring them. Moreover, ensembling over individual models was robust to hidden random annotators. In blind prospective tests of 52,555 subsequent expert-annotated images, the models labeled pathologies like their human counterparts (consensus model AUPRC=0.74 cored; 0.69 CAA). This study demonstrates a means to combine multiple ground truths into a common-ground DL model that yields consistent diagnoses informed by multiple and potentially variable expert opinion.

Competing Interest Statement

The authors have declared no competing interest.

Footnotes

  • https://github.com/keiserlab/consensus-learning-paper

  • https://osf.io/xh2jd/

  • List of Abbreviations

    DL
    deep learning
    CAA
    cerebral amyloid angiopathy
    Aβ
    amyloid beta
    WSIs
    whole slide images
    IRB
    Institutional Review Board
    HSV
    hue saturation value
    CNN
    convolutional neural network
    NP#
    neuropathologist #
    UG#
    undergraduate novice #
    AUROC
    area under the receiver operating characteristic
    AUPRC
    area under the precision recall curve
    CAMs
    class activation maps
  • Copyright 
    The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY 4.0 International license.
    Back to top
    PreviousNext
    Posted April 10, 2022.
    Download PDF

    Supplementary Material

    Data/Code
    Email

    Thank you for your interest in spreading the word about bioRxiv.

    NOTE: Your email address is requested solely to identify you as the sender of this article.

    Enter multiple addresses on separate lines or separate them with commas.
    Deep learning from multiple experts improves identification of amyloid neuropathologies
    (Your Name) has forwarded a page to you from bioRxiv
    (Your Name) thought you would like to see this page from the bioRxiv website.
    CAPTCHA
    This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
    Share
    Deep learning from multiple experts improves identification of amyloid neuropathologies
    Daniel R. Wong, Ziqi Tang, Nicholas C. Mew, Sakshi Das, Justin Athey, Kirsty E. McAleese, Julia K. Kofler, Margaret E. Flanagan, Ewa Borys, Charles L. White III, Atul J. Butte, Brittany N. Dugger, Michael J. Keiser
    bioRxiv 2021.03.12.435050; doi: https://doi.org/10.1101/2021.03.12.435050
    Digg logo Reddit logo Twitter logo Facebook logo Google logo LinkedIn logo Mendeley logo
    Citation Tools
    Deep learning from multiple experts improves identification of amyloid neuropathologies
    Daniel R. Wong, Ziqi Tang, Nicholas C. Mew, Sakshi Das, Justin Athey, Kirsty E. McAleese, Julia K. Kofler, Margaret E. Flanagan, Ewa Borys, Charles L. White III, Atul J. Butte, Brittany N. Dugger, Michael J. Keiser
    bioRxiv 2021.03.12.435050; doi: https://doi.org/10.1101/2021.03.12.435050

    Citation Manager Formats

    • BibTeX
    • Bookends
    • EasyBib
    • EndNote (tagged)
    • EndNote 8 (xml)
    • Medlars
    • Mendeley
    • Papers
    • RefWorks Tagged
    • Ref Manager
    • RIS
    • Zotero
    • Tweet Widget
    • Facebook Like
    • Google Plus One

    Subject Area

    • Pathology
    Subject Areas
    All Articles
    • Animal Behavior and Cognition (3514)
    • Biochemistry (7365)
    • Bioengineering (5342)
    • Bioinformatics (20318)
    • Biophysics (10041)
    • Cancer Biology (7773)
    • Cell Biology (11348)
    • Clinical Trials (138)
    • Developmental Biology (6450)
    • Ecology (9979)
    • Epidemiology (2065)
    • Evolutionary Biology (13354)
    • Genetics (9370)
    • Genomics (12607)
    • Immunology (7724)
    • Microbiology (19087)
    • Molecular Biology (7459)
    • Neuroscience (41134)
    • Paleontology (300)
    • Pathology (1235)
    • Pharmacology and Toxicology (2142)
    • Physiology (3177)
    • Plant Biology (6878)
    • Scientific Communication and Education (1276)
    • Synthetic Biology (1900)
    • Systems Biology (5328)
    • Zoology (1091)