Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media

View ORCID ProfileAndrew J. Schaumberg, View ORCID ProfileWendy C. Juarez-Nicanor, View ORCID ProfileSarah J. Choudhury, View ORCID ProfileLaura G. Pastrián, View ORCID ProfileBobbi S. Pritt, Mario Prieto Pozuelo, Ricardo Sotillo Sánchez, Khanh Ho, Nusrat Zahra, View ORCID ProfileBetul Duygu Sener, View ORCID ProfileStephen Yip, View ORCID ProfileBin Xu, View ORCID ProfileSrinivas Rao Annavarapu, Aurélien Morini, View ORCID ProfileKarra A. Jones, Kathia Rosado-Orozco, View ORCID ProfileSanjay Mukhopadhyay, View ORCID ProfileCarlos Miguel, Hongyu Yang, Yale Rosen, View ORCID ProfileRola H. Ali, View ORCID ProfileOlaleke O. Folaranmi, View ORCID ProfileJerad M. Gardner, Corina Rusu, View ORCID ProfileCelina Stayerman, View ORCID ProfileJohn Gross, Dauda E. Suleiman, View ORCID ProfileS. Joseph Sirintrapun, View ORCID ProfileMariam Aly, View ORCID ProfileThomas J. Fuchs
doi: https://doi.org/10.1101/396663
Andrew J. Schaumberg
1Memorial Sloan Kettering Cancer Center and the Tri-Institutional Training Program in Computational Biology and Medicine, NY, USA
2Weill Cornell Graduate School of Medical Sciences, NY, USA
3Weill Cornell High School Science Immersion Program
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Andrew J. Schaumberg
  • For correspondence: ajs625@cornell.edu
Wendy C. Juarez-Nicanor
3Weill Cornell High School Science Immersion Program
4Manhattan/Hunter Science High School, NY, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Wendy C. Juarez-Nicanor
Sarah J. Choudhury
3Weill Cornell High School Science Immersion Program
4Manhattan/Hunter Science High School, NY, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Sarah J. Choudhury
Laura G. Pastrián
5Hospital Universitario La Paz, Department of Pathology, Madrid, Spain
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Laura G. Pastrián
Bobbi S. Pritt
6Mayo Clinic, Department of Laboratory Medicine and Pathology, MN, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Bobbi S. Pritt
Mario Prieto Pozuelo
7Hospital Universitario HM Sanchinarro, Laboratorio de Dianas Terapéuticas, Madrid, Spain
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ricardo Sotillo Sánchez
8Virgen de Altagracia Hospital, Departamento de Patología, Spain
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Khanh Ho
9Centre Hospitalier de Mouscron, Département de Pathologie. Belgium
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Nusrat Zahra
10Allama Iqbal Medical College, Department of Pathology, Lahore, Pakistan
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Betul Duygu Sener
11Konya Training and Research Hospital, Department of Pathology, Konya, Turkey
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Betul Duygu Sener
Stephen Yip
12BC Cancer, Department of Pathology, British Columbia, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Stephen Yip
Bin Xu
13Sunnybrook Health Sciences Centre, Department of Pathology, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Bin Xu
Srinivas Rao Annavarapu
14Royal Victoria Infirmary, Department of Cellular Pathology, England, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Srinivas Rao Annavarapu
Aurélien Morini
15Université Paris Est Créteil, Faculté de médecine de Créteil, France
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Karra A. Jones
16University of Iowa, Department of Pathology, IA, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Karra A. Jones
Kathia Rosado-Orozco
17HRP Labs, San Juan, Puerto Rico, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sanjay Mukhopadhyay
18Cleveland Clinic, Department of Pathology, Cleveland, OH, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Sanjay Mukhopadhyay
Carlos Miguel
19Centro Médico de Asturias, Department of Pathology, Oviedo, Spain
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Carlos Miguel
Hongyu Yang
20St Vincent Evansville Hospital, Department of Pathology, Evansville, IN, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Yale Rosen
21SUNY Downstate Medical Center, Department of Pathology, NY, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Rola H. Ali
22Kuwait University, Faculty of Medicine, Kuwait
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Rola H. Ali
Olaleke O. Folaranmi
23University of Ilorin Teaching Hospital, Department of Pathology, Nigeria
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Olaleke O. Folaranmi
Jerad M. Gardner
24University of Arkansas for Medical Sciences, Department of Pathology, Little Rock, AK, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Jerad M. Gardner
Corina Rusu
25Augusta Hospital, Department of Pathology, Bochum, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Celina Stayerman
26Laboratorio TechniPath, San Pedro Sula, Honduras
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Celina Stayerman
John Gross
27Mayo Clinic, Bone and Soft Tissue and Surgical Pathology, Rochester, MN, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for John Gross
Dauda E. Suleiman
28Abubakar Tafawa Balewa University Teaching Hospital, Department of Histopathology, Bauchi, Nigeria
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
S. Joseph Sirintrapun
29Memorial Sloan Kettering Cancer Center, Department of Pathology, NY, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for S. Joseph Sirintrapun
Mariam Aly
30Columbia University, Department of Psychology, NY, USA
31Affiliate Member of the Zuckerman Mind Brain Behavior Institute, Columbia University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Mariam Aly
  • For correspondence: ajs625@cornell.edu
Thomas J. Fuchs
2Weill Cornell Graduate School of Medical Sciences, NY, USA
29Memorial Sloan Kettering Cancer Center, Department of Pathology, NY, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Thomas J. Fuchs
  • For correspondence: ajs625@cornell.edu
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Data/Code
  • Preview PDF
Loading

Abstract

Pathologists are responsible for rapidly providing a diagnosis on critical health issues. Challenging cases benefit from additional opinions of pathologist colleagues. In addition to on-site colleagues, there is an active worldwide community of pathologists on social media for complementary opinions. Such access to pathologists worldwide has the capacity to improve diagnostic accuracy and generate broader consensus on next steps in patient care. From Twitter we curate 13,626 images from 6,351 tweets from 25 pathologists from 13 countries. We supplement the Twitter data with 113,161 images from 1,074,484 PubMed articles. We develop machine learning and deep learning models to (i) accurately identify histopathology stains, (ii) discriminate between tissues, and (iii) differentiate disease states. Area Under Receiver Operating Characteristic is 0.805-0.996 for these tasks. We repurpose the disease classifier to search for similar disease states given an image and clinical covariates. We report precision@k=1 = 0.7618±0.0018 (chance 0.397±0.004, mean±stdev). The classifiers find texture and tissue are important clinico-visual features of disease. Deep features trained only on natural images (e.g. cats and dogs) substantially improved search performance, while pathology-specific deep features and cell nuclei features further improved search to a lesser extent. We implement a social media bot (@pathobot on Twitter) to use the trained classifiers to aid pathologists in obtaining real-time feedback on challenging cases. If a social media post containing pathology text and images mentions the bot, the bot generates quantitative predictions of disease state (normal/artifact/infection/injury/nontumor, pre-neoplastic/benign/ low-grade-malignant-potential, or malignant) and lists similar cases across social media and PubMed. Our project has become a globally distributed expert system that facilitates pathological diagnosis and brings expertise to underserved regions or hospitals with less expertise in a particular disease. This is the first pan-tissue pan-disease (i.e. from infection to malignancy) method for prediction and search on social media, and the first pathology study prospectively tested in public on social media. We will share data through pathobotology.org. We expect our project to cultivate a more connected world of physicians and improve patient care worldwide.

Footnotes

  • ↵β These pathologist authors generously donated cases.

  • ↵δ These authors are Principal Investigators of this work.

  • ↵a ajs625{at}cornell.edu (Twitter @schaumberg_a)

  • ↵b ma3631{at}columbia.edu (Twitter @mariam_s_aly)

  • ↵c fuchst{at}mskcc.org (Twitter @ThomasFuchsAI).

  • Link to site to host data http://pathobotology.org Additional cluster and importance analyses. Improved performance.

  • https://twitter.com/pathobot

  • http://pathobotology.org

  • ↵1 ImageIO documentation available here: https://docs.oracle.com/javase/7/docs/api/javax/imageio/Image10.html

  • ↵2 Courts in the United States have ruled that images posted to social media are still owned by their authors and are not public domain. Indeed, in Morel v. AFP, AFP was ordered to pay Morel $1.2 million for copyright infringement because AFP used images that Morel posted to social media.

  • ↵3 Normal cerebellum case by S.Y. at https://twitter.com/Sty_md/status/821840894634565632

  • ↵4 A case of this is from author K.H., where a different pathologist gave the diagnosis, and he agreed. We summarized this as “metastatic lobular carcinoma” in the auxiliary annotation file for the tweet https://twitter.com/Ho_Khanh_MD/status/999989201734197250.

  • ↵5 A case of this is from author M.P.P., where M.P.P. wrote “IDC DIN LISN” directly on a shared histology image in the tweet https://twitter.com/dr_MPrieto/status/890118713155997696 so we wrote this text in the auxiliary annotation file for the tweet.

  • ↵6 A case of this is from K.H., observing iron pill lesions in stomach biopsy https://twitter.com/Ho_Khanh_MD/status/963800933716123648.

  • ↵7 For this formula please see https://github.com/keras-team/keras/issues/6444

  • ↵8 Case at https://twitter.com/BinXu16/status/980404471833313280 “Kudo to @drkennethtang @luishcruzc and @DrGeeONE The answer of this case can be seen in the right corner of the 3rd picture. Dx: Echinococcus (hydatid cyst) with necrotizing pneumonia, abscess, and granulomatous inflammation. Additional high power pictures attached.”

Copyright 
The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license.
Back to top
PreviousNext
Posted March 09, 2020.
Download PDF
Data/Code
Email

Thank you for your interest in spreading the word about bioRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media
(Your Name) has forwarded a page to you from bioRxiv
(Your Name) thought you would like to see this page from the bioRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media
Andrew J. Schaumberg, Wendy C. Juarez-Nicanor, Sarah J. Choudhury, Laura G. Pastrián, Bobbi S. Pritt, Mario Prieto Pozuelo, Ricardo Sotillo Sánchez, Khanh Ho, Nusrat Zahra, Betul Duygu Sener, Stephen Yip, Bin Xu, Srinivas Rao Annavarapu, Aurélien Morini, Karra A. Jones, Kathia Rosado-Orozco, Sanjay Mukhopadhyay, Carlos Miguel, Hongyu Yang, Yale Rosen, Rola H. Ali, Olaleke O. Folaranmi, Jerad M. Gardner, Corina Rusu, Celina Stayerman, John Gross, Dauda E. Suleiman, S. Joseph Sirintrapun, Mariam Aly, Thomas J. Fuchs
bioRxiv 396663; doi: https://doi.org/10.1101/396663
Digg logo Reddit logo Twitter logo Facebook logo Google logo LinkedIn logo Mendeley logo
Citation Tools
Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media
Andrew J. Schaumberg, Wendy C. Juarez-Nicanor, Sarah J. Choudhury, Laura G. Pastrián, Bobbi S. Pritt, Mario Prieto Pozuelo, Ricardo Sotillo Sánchez, Khanh Ho, Nusrat Zahra, Betul Duygu Sener, Stephen Yip, Bin Xu, Srinivas Rao Annavarapu, Aurélien Morini, Karra A. Jones, Kathia Rosado-Orozco, Sanjay Mukhopadhyay, Carlos Miguel, Hongyu Yang, Yale Rosen, Rola H. Ali, Olaleke O. Folaranmi, Jerad M. Gardner, Corina Rusu, Celina Stayerman, John Gross, Dauda E. Suleiman, S. Joseph Sirintrapun, Mariam Aly, Thomas J. Fuchs
bioRxiv 396663; doi: https://doi.org/10.1101/396663

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Pathology
Subject Areas
All Articles
  • Animal Behavior and Cognition (4087)
  • Biochemistry (8762)
  • Bioengineering (6479)
  • Bioinformatics (23341)
  • Biophysics (11750)
  • Cancer Biology (9149)
  • Cell Biology (13248)
  • Clinical Trials (138)
  • Developmental Biology (7417)
  • Ecology (11369)
  • Epidemiology (2066)
  • Evolutionary Biology (15087)
  • Genetics (10399)
  • Genomics (14009)
  • Immunology (9121)
  • Microbiology (22040)
  • Molecular Biology (8779)
  • Neuroscience (47368)
  • Paleontology (350)
  • Pathology (1420)
  • Pharmacology and Toxicology (2482)
  • Physiology (3704)
  • Plant Biology (8050)
  • Scientific Communication and Education (1431)
  • Synthetic Biology (2208)
  • Systems Biology (6016)
  • Zoology (1249)