Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

Outcomes of the 2019 EMDataResource model challenge: validation of cryo-EM models at near-atomic resolution

View ORCID ProfileCatherine L. Lawson, View ORCID ProfileAndriy Kryshtafovych, View ORCID ProfilePaul D. Adams, View ORCID ProfilePavel V. Afonine, View ORCID ProfileMatthew L. Baker, View ORCID ProfileBenjamin A. Barad, View ORCID ProfilePaul Bond, View ORCID ProfileTom Burnley, View ORCID ProfileRenzhi Cao, View ORCID ProfileJianlin Cheng, View ORCID ProfileGrzegorz Chojnowski, View ORCID ProfileKevin Cowtan, View ORCID ProfileKen A. Dill, View ORCID ProfileFrank DiMaio, View ORCID ProfileDaniel P. Farrell, View ORCID ProfileJames S. Fraser, View ORCID ProfileMark A. Herzik Jr., View ORCID ProfileSoon Wen Hoh, View ORCID ProfileJie Hou, View ORCID ProfileLi-Wei Hung, View ORCID ProfileMaxim Igaev, View ORCID ProfileAgnel P. Joseph, View ORCID ProfileDaisuke Kihara, View ORCID ProfileDilip Kumar, Sumit Mittal, View ORCID ProfileBohdan Monastyrskyy, View ORCID ProfileMateusz Olek, View ORCID ProfileColin M. Palmer, View ORCID ProfileArdan Patwardhan, View ORCID ProfileAlberto Perez, View ORCID ProfileJonas Pfab, View ORCID ProfileGrigore D. Pintilie, View ORCID ProfileJane S. Richardson, View ORCID ProfilePeter B. Rosenthal, View ORCID ProfileDaipayan Sarkar, View ORCID ProfileLuisa U. Schäfer, View ORCID ProfileMichael F. Schmid, View ORCID ProfileGunnar F. Schröder, View ORCID ProfileMrinal Shekhar, View ORCID ProfileDong Si, View ORCID ProfileAbishek Singharoy, View ORCID ProfileGenki Terashi, View ORCID ProfileThomas C. Terwilliger, View ORCID ProfileAndrea Vaiana, Liguo Wang, View ORCID ProfileZhe Wang, View ORCID ProfileStephanie A. Wankowicz, View ORCID ProfileChristopher J. Williams, View ORCID ProfileMartyn Winn, View ORCID ProfileTianqi Wu, View ORCID ProfileXiaodi Yu, View ORCID ProfileKaiming Zhang, View ORCID ProfileHelen M. Berman, View ORCID ProfileWah Chiu
doi: https://doi.org/10.1101/2020.06.12.147033
Catherine L. Lawson
1Institute for Quantitative Biomedicine and Research Collaboratory for Structural Bioinformatics, Rutgers, The State University of New Jersey, Piscataway, New Jersey 08854, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Catherine L. Lawson
  • For correspondence: cathy.lawson@rutgers.edu wahc@stanford.edu
Andriy Kryshtafovych
2Genome Center, University of California, Davis, California 95616, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Andriy Kryshtafovych
Paul D. Adams
3Molecular Biophysics & Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, 94720, USA
4Department of Bioengineering, University of California Berkeley, Berkeley, CA 94720, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Paul D. Adams
Pavel V. Afonine
3Molecular Biophysics & Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, 94720, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Pavel V. Afonine
Matthew L. Baker
5The University of Texas Health Science Center at Houston, Department of Biochemistry and Molecular Biology, Houston, TX 77030
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Matthew L. Baker
Benjamin A. Barad
6The Scripps Research Institute, Department of Integrated Computational Structural Biology, La Jolla, CA 92103
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Benjamin A. Barad
Paul Bond
7York Structural Biology Laboratory, Department of Chemistry, University of York, York YO10 5DD, England, U.K.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Paul Bond
Tom Burnley
8Scientific Computing Department, UKRI Science and Technology Facilities Council, Research Complex at Harwell, Didcot OX11 0FA, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Tom Burnley
Renzhi Cao
9Department of Computer Science, Pacific Lutheran University, Tacoma, WA 98447, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Renzhi Cao
Jianlin Cheng
10Department of Electrical Engineering and Computer Science, University of Missouri, Columbia, MO 65211, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Jianlin Cheng
Grzegorz Chojnowski
11European Molecular Biology Laboratory, c/o DESY, Notkestrasse 85, 22607 Hamburg, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Grzegorz Chojnowski
Kevin Cowtan
7York Structural Biology Laboratory, Department of Chemistry, University of York, York YO10 5DD, England, U.K.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Kevin Cowtan
Ken A. Dill
12Laufer Center, Stony Brook University, Stony Brook, New York, 11794, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Ken A. Dill
Frank DiMaio
13Department of Biochemistry and Institute for Protein Design, University of Washington, Seattle, WA 98195, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Frank DiMaio
Daniel P. Farrell
13Department of Biochemistry and Institute for Protein Design, University of Washington, Seattle, WA 98195, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Daniel P. Farrell
James S. Fraser
14Department of Bioengineering and Therapeutic Sciences, University of California San Francisco, San Francisco, CA 94158, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for James S. Fraser
Mark A. Herzik Jr.
15Department of Chemistry and Biochemistry, University of California, San Diego, 9500 Gilman Drive, La Jolla, CA 92093, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Mark A. Herzik Jr.
Soon Wen Hoh
7York Structural Biology Laboratory, Department of Chemistry, University of York, York YO10 5DD, England, U.K.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Soon Wen Hoh
Jie Hou
16Department of Computer Science, Saint Louis University, St. Louis, MO, 63103, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Jie Hou
Li-Wei Hung
17Los Alamos National Laboratory, Los Alamos, NM 87545, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Li-Wei Hung
Maxim Igaev
18Theoretical and Computational Biophysics, Max Planck Institute for Biophysical Chemistry, D-37077 Göttingen, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Maxim Igaev
Agnel P. Joseph
8Scientific Computing Department, UKRI Science and Technology Facilities Council, Research Complex at Harwell, Didcot OX11 0FA, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Agnel P. Joseph
Daisuke Kihara
19Department of Biological Sciences, Purdue University, West Lafayette, IN, 47907, USA
20Department of Computer Science, Purdue University, West Lafayette, IN, 47907, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Daisuke Kihara
Dilip Kumar
21Verna and Marrs McLean Department of Biochemistry and Molecular Biology, Baylor College of Medicine, Houston, TX 77030, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Dilip Kumar
Sumit Mittal
22Biodesign Institute, Arizona State University, Tempe, AZ, 85201, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Bohdan Monastyrskyy
2Genome Center, University of California, Davis, California 95616, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Bohdan Monastyrskyy
Mateusz Olek
7York Structural Biology Laboratory, Department of Chemistry, University of York, York YO10 5DD, England, U.K.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Mateusz Olek
Colin M. Palmer
8Scientific Computing Department, UKRI Science and Technology Facilities Council, Research Complex at Harwell, Didcot OX11 0FA, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Colin M. Palmer
Ardan Patwardhan
23The European Bioinformatics Institute (EMBL-EBI), Wellcome Genome Campus, Hinxton, Cambridgeshire, United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Ardan Patwardhan
Alberto Perez
24Department of Chemistry, University of Florida, Gainesville, Florida, 32611, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Alberto Perez
Jonas Pfab
25Division of Computing & Software Systems, University of Washington, Bothell, WA, 98011, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Jonas Pfab
Grigore D. Pintilie
26Department of Bioengineering, Stanford University, Stanford, California 94305, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Grigore D. Pintilie
Jane S. Richardson
27Department of Biochemistry, Duke University, Durham NC 27710, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Jane S. Richardson
Peter B. Rosenthal
28Structural Biology of Cells and Viruses Laboratory, Francis Crick Institute, London, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Peter B. Rosenthal
Daipayan Sarkar
19Department of Biological Sciences, Purdue University, West Lafayette, IN, 47907, USA
22Biodesign Institute, Arizona State University, Tempe, AZ, 85201, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Daipayan Sarkar
Luisa U. Schäfer
29Institute of Biological Information Processing (IBI-7: Structural Biochemistry) and Jülich Centre for Structural Biology (JuStruct), Forschungszentrum Jülich, Jülich, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Luisa U. Schäfer
Michael F. Schmid
30Division of CryoEM and Biomaging, SSRL, SLAC National Accelerator Laboratory, Stanford University, Menlo Park, California 94025, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Michael F. Schmid
Gunnar F. Schröder
29Institute of Biological Information Processing (IBI-7: Structural Biochemistry) and Jülich Centre for Structural Biology (JuStruct), Forschungszentrum Jülich, Jülich, Germany
31Physics Department, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Gunnar F. Schröder
Mrinal Shekhar
22Biodesign Institute, Arizona State University, Tempe, AZ, 85201, USA
32Center for Development of Therapeutics, Broad Institute of MIT and Harvard, Cambridge, MA, 02141, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Mrinal Shekhar
Dong Si
25Division of Computing & Software Systems, University of Washington, Bothell, WA, 98011, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Dong Si
Abishek Singharoy
22Biodesign Institute, Arizona State University, Tempe, AZ, 85201, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Abishek Singharoy
Genki Terashi
18Theoretical and Computational Biophysics, Max Planck Institute for Biophysical Chemistry, D-37077 Göttingen, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Genki Terashi
Thomas C. Terwilliger
33New Mexico Consortium, Los Alamos NM 87544, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Thomas C. Terwilliger
Andrea Vaiana
18Theoretical and Computational Biophysics, Max Planck Institute for Biophysical Chemistry, D-37077 Göttingen, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Andrea Vaiana
Liguo Wang
34Department of Biological Structure, University of Washington, Seattle, WA 98195, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Zhe Wang
23The European Bioinformatics Institute (EMBL-EBI), Wellcome Genome Campus, Hinxton, Cambridgeshire, United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Zhe Wang
Stephanie A. Wankowicz
14Department of Bioengineering and Therapeutic Sciences, University of California San Francisco, San Francisco, CA 94158, USA
35Biophysics Graduate Program, University of California, San Francisco, CA 94158, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Stephanie A. Wankowicz
Christopher J. Williams
27Department of Biochemistry, Duke University, Durham NC 27710, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Christopher J. Williams
Martyn Winn
8Scientific Computing Department, UKRI Science and Technology Facilities Council, Research Complex at Harwell, Didcot OX11 0FA, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Martyn Winn
Tianqi Wu
36Department of Electrical Engineering and Computer Science, University of Missouri, Columbia, MO 65211, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Tianqi Wu
Xiaodi Yu
37SMPS, Janssen Research and Development, 1400 McKean Rd, Spring House, PA, 19477, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Xiaodi Yu
Kaiming Zhang
26Department of Bioengineering, Stanford University, Stanford, California 94305, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Kaiming Zhang
Helen M. Berman
38Department of Chemistry and Chemical Biology and Research Collaboratory for Structural Bioinformatics, Rutgers, The State University of New Jersey, Piscataway, New Jersey 08854, USA
39Department of Biological Sciences and Bridge Institute, University of Southern California, Los Angeles, California 90089, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Helen M. Berman
Wah Chiu
26Department of Bioengineering, Stanford University, Stanford, California 94305, USA
30Division of CryoEM and Biomaging, SSRL, SLAC National Accelerator Laboratory, Stanford University, Menlo Park, California 94025, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Wah Chiu
  • For correspondence: cathy.lawson@rutgers.edu wahc@stanford.edu
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Supplementary material
  • Data/Code
  • Preview PDF
Loading

Abstract

This paper describes outcomes of the 2019 Cryo-EM Map-based Model Metrics Challenge sponsored by EMDataResource (www.emdataresource.org). The goals of this challenge were (1) to assess the quality of models that can be produced using current modeling software, (2) to check the reproducibility of modeling results from different software developers and users, and (3) compare the performance of current metrics used for evaluation of models. The focus was on near-atomic resolution maps with an innovative twist: three of four target maps formed a resolution series (1.8 to 3.1 Å) from the same specimen and imaging experiment. Tools developed in previous challenges were expanded for managing, visualizing and analyzing the 63 submitted coordinate models, and several novel metrics were introduced. The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual laboratory experiments and holdings of structure data archives such as the Protein Data Bank. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived from these benchmark maps by 13 participating teams, representing both widely used and novel modeling approaches. We also evaluate the pros and cons of the commonly used metrics to assess model quality and recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed density in the cryo-EM map.

Competing Interest Statement

The authors have declared no competing interest.

Footnotes

  • https://model-compare.emdataresource.org/

  • https://challenges.emdataresource.org/

Copyright 
The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-ND 4.0 International license.
Back to top
PreviousNext
Posted June 15, 2020.
Download PDF

Supplementary Material

Data/Code
Email

Thank you for your interest in spreading the word about bioRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Outcomes of the 2019 EMDataResource model challenge: validation of cryo-EM models at near-atomic resolution
(Your Name) has forwarded a page to you from bioRxiv
(Your Name) thought you would like to see this page from the bioRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Outcomes of the 2019 EMDataResource model challenge: validation of cryo-EM models at near-atomic resolution
Catherine L. Lawson, Andriy Kryshtafovych, Paul D. Adams, Pavel V. Afonine, Matthew L. Baker, Benjamin A. Barad, Paul Bond, Tom Burnley, Renzhi Cao, Jianlin Cheng, Grzegorz Chojnowski, Kevin Cowtan, Ken A. Dill, Frank DiMaio, Daniel P. Farrell, James S. Fraser, Mark A. Herzik Jr., Soon Wen Hoh, Jie Hou, Li-Wei Hung, Maxim Igaev, Agnel P. Joseph, Daisuke Kihara, Dilip Kumar, Sumit Mittal, Bohdan Monastyrskyy, Mateusz Olek, Colin M. Palmer, Ardan Patwardhan, Alberto Perez, Jonas Pfab, Grigore D. Pintilie, Jane S. Richardson, Peter B. Rosenthal, Daipayan Sarkar, Luisa U. Schäfer, Michael F. Schmid, Gunnar F. Schröder, Mrinal Shekhar, Dong Si, Abishek Singharoy, Genki Terashi, Thomas C. Terwilliger, Andrea Vaiana, Liguo Wang, Zhe Wang, Stephanie A. Wankowicz, Christopher J. Williams, Martyn Winn, Tianqi Wu, Xiaodi Yu, Kaiming Zhang, Helen M. Berman, Wah Chiu
bioRxiv 2020.06.12.147033; doi: https://doi.org/10.1101/2020.06.12.147033
Reddit logo Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools
Outcomes of the 2019 EMDataResource model challenge: validation of cryo-EM models at near-atomic resolution
Catherine L. Lawson, Andriy Kryshtafovych, Paul D. Adams, Pavel V. Afonine, Matthew L. Baker, Benjamin A. Barad, Paul Bond, Tom Burnley, Renzhi Cao, Jianlin Cheng, Grzegorz Chojnowski, Kevin Cowtan, Ken A. Dill, Frank DiMaio, Daniel P. Farrell, James S. Fraser, Mark A. Herzik Jr., Soon Wen Hoh, Jie Hou, Li-Wei Hung, Maxim Igaev, Agnel P. Joseph, Daisuke Kihara, Dilip Kumar, Sumit Mittal, Bohdan Monastyrskyy, Mateusz Olek, Colin M. Palmer, Ardan Patwardhan, Alberto Perez, Jonas Pfab, Grigore D. Pintilie, Jane S. Richardson, Peter B. Rosenthal, Daipayan Sarkar, Luisa U. Schäfer, Michael F. Schmid, Gunnar F. Schröder, Mrinal Shekhar, Dong Si, Abishek Singharoy, Genki Terashi, Thomas C. Terwilliger, Andrea Vaiana, Liguo Wang, Zhe Wang, Stephanie A. Wankowicz, Christopher J. Williams, Martyn Winn, Tianqi Wu, Xiaodi Yu, Kaiming Zhang, Helen M. Berman, Wah Chiu
bioRxiv 2020.06.12.147033; doi: https://doi.org/10.1101/2020.06.12.147033

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Biochemistry
Subject Areas
All Articles
  • Animal Behavior and Cognition (4369)
  • Biochemistry (9543)
  • Bioengineering (7068)
  • Bioinformatics (24765)
  • Biophysics (12559)
  • Cancer Biology (9923)
  • Cell Biology (14296)
  • Clinical Trials (138)
  • Developmental Biology (7929)
  • Ecology (12073)
  • Epidemiology (2067)
  • Evolutionary Biology (15952)
  • Genetics (10901)
  • Genomics (14704)
  • Immunology (9841)
  • Microbiology (23580)
  • Molecular Biology (9453)
  • Neuroscience (50691)
  • Paleontology (369)
  • Pathology (1535)
  • Pharmacology and Toxicology (2674)
  • Physiology (3996)
  • Plant Biology (8638)
  • Scientific Communication and Education (1505)
  • Synthetic Biology (2388)
  • Systems Biology (6413)
  • Zoology (1344)