RT Journal Article
SR Electronic
T1 Variability in the analysis of a single neuroimaging dataset by many teams
JF bioRxiv
FD Cold Spring Harbor Laboratory
SP 843193
DO 10.1101/843193
A1 Botvinik-Nezer, Rotem
A1 Holzmeister, Felix
A1 Camerer, Colin F.
A1 Dreber, Anna
A1 Huber, Juergen
A1 Johannesson, Magnus
A1 Kirchler, Michael
A1 Iwanir, Roni
A1 Mumford, Jeanette A.
A1 Adcock, Alison
A1 Avesani, Paolo
A1 Baczkowski, Blazej
A1 Bajracharya, Aahana
A1 Bakst, Leah
A1 Ball, Sheryl
A1 Barilari, Marco
A1 Bault, Nadège
A1 Beaton, Derek
A1 Beitner, Julia
A1 Benoit, Roland
A1 Berkers, Ruud
A1 Bhanji, Jamil
A1 Biswal, Bharat
A1 Bobadilla-Suarez, Sebastian
A1 Bortolini, Tiago
A1 Bottenhorn, Katherine
A1 Bowring, Alexander
A1 Braem, Senne
A1 Brooks, Hayley
A1 Brudner, Emily
A1 Calderon, Cristian
A1 Camilleri, Julia
A1 Castrellon, Jaime
A1 Cecchetti, Luca
A1 Cieslik, Edna
A1 Cole, Zachary
A1 Collignon, Olivier
A1 Cox, Robert
A1 Cunningham, William
A1 Czoschke, Stefan
A1 Dadi, Kamalaker
A1 Davis, Charles
A1 De Luca, Alberto
A1 Delgado, Mauricio
A1 Demetriou, Lysia
A1 Dennison, Jeffrey
A1 Di, Xin
A1 Dickie, Erin
A1 Dobryakova, Ekaterina
A1 Donnat, Claire
A1 Dukart, Juergen
A1 Duncan, Niall W.
A1 Durnez, Joke
A1 Eed, Amr
A1 Eickhoff, Simon
A1 Erhart, Andrew
A1 Fontanesi, Laura
A1 Fricke, G. Matthew
A1 Galvan, Adriana
A1 Gau, Remi
A1 Genon, Sarah
A1 Glatard, Tristan
A1 Glerean, Enrico
A1 Goeman, Jelle
A1 Golowin, Sergej
A1 González-García, Carlos
A1 Gorgolewski, Krzysztof
A1 Grady, Cheryl
A1 Green, Mikella
A1 Guassi Moreira, João
A1 Guest, Olivia
A1 Hakimi, Shabnam
A1 Hamilton, J. Paul
A1 Hancock, Roeland
A1 Handjaras, Giacomo
A1 Harry, Bronson
A1 Hawco, Colin
A1 Herholz, Peer
A1 Herman, Gabrielle
A1 Heunis, Stephan
A1 Hoffstaedter, Felix
A1 Hogeveen, Jeremy
A1 Holmes, Susan
A1 Hu, Chuan-Peng
A1 Huettel, Scott
A1 Hughes, Matthew
A1 Iacovella, Vittorio
A1 Iordan, Alexandru
A1 Isager, Peder
A1 Isik, Ayse Ilkay
A1 Jahn, Andrew
A1 Johnson, Matthew
A1 Johnstone, Tom
A1 Joseph, Michael
A1 Juliano, Anthony
A1 Kable, Joseph
A1 Kassinopoulos, Michalis
A1 Koba, Cemal
A1 Kong, Xiang-Zhen
A1 Koscik, Timothy
A1 Kucukboyaci, Nuri Erkut
A1 Kuhl, Brice
A1 Kupek, Sebastian
A1 Laird, Angela
A1 Lamm, Claus
A1 Langner, Robert
A1 Lauharatanahirun, Nina
A1 Lee, Hongmi
A1 Lee, Sangil
A1 Leemans, Alexander
A1 Leo, Andrea
A1 Lesage, Elise
A1 Li, Flora
A1 Li, Monica
A1 Lim, Phui Cheng
A1 Lintz, Evan
A1 Liphardt, Schuyler
A1 Losecaat Vermeer, Annabel
A1 Love, Bradley
A1 Mack, Michael
A1 Malpica, Norberto
A1 Marins, Theo
A1 Maumet, Camille
A1 McDonald, Kelsey
A1 McGuire, Joseph
A1 Melero, Helena
A1 Méndez Leal, Adriana
A1 Meyer, Benjamin
A1 Meyer, Kristin
A1 Mihai, Paul
A1 Mitsis, Georgios
A1 Moll, Jorge
A1 Nielson, Dylan
A1 Nilsonne, Gustav
A1 Notter, Michael
A1 Olivetti, Emanuele
A1 Onicas, Adrian
A1 Papale, Paolo
A1 Patil, Kaustubh
A1 Peelle, Jonathan E.
A1 Pérez, Alexandre
A1 Pischedda, Doris
A1 Poline, Jean-Baptiste
A1 Prystauka, Yanina
A1 Ray, Shruti
A1 Reuter-Lorenz, Patricia
A1 Reynolds, Richard
A1 Ricciardi, Emiliano
A1 Rieck, Jenny
A1 Rodriguez-Thompson, Anais
A1 Romyn, Anthony
A1 Salo, Taylor
A1 Samanez-Larkin, Gregory
A1 Sanz-Morales, Emilio
A1 Schlichting, Margaret
A1 Schultz, Douglas
A1 Shen, Qiang
A1 Sheridan, Margaret
A1 Shiguang, Fu
A1 Silvers, Jennifer
A1 Skagerlund, Kenny
A1 Smith, Alec
A1 Smith, David
A1 Sokol-Hessner, Peter
A1 Steinkamp, Simon
A1 Tashjian, Sarah
A1 Thirion, Bertrand
A1 Thorp, John
A1 Tinghög, Gustav
A1 Tisdall, Loreen
A1 Tompson, Steven
A1 Toro-Serey, Claudio
A1 Torre, Juan
A1 Tozzi, Leonardo
A1 Truong, Vuong
A1 Turella, Luca
A1 van’t Veer, Anna E.
A1 Verguts, Tom
A1 Vettel, Jean
A1 Vijayarajah, Sagana
A1 Vo, Khoi
A1 Wall, Matthew
A1 Weeda, Wouter D.
A1 Weis, Susanne
A1 White, David
A1 Wisniewski, David
A1 Xifra-Porxas, Alba
A1 Yearling, Emily
A1 Yoon, Sangsuk
A1 Yuan, Rui
A1 Yuen, Kenneth
A1 Zhang, Lei
A1 Zhang, Xu
A1 Zosky, Joshua
A1 Nichols, Thomas E.
A1 Poldrack, Russell A.
A1 Schonberg, Tom
YR 2019
UL http://biorxiv.org/content/early/2019/11/15/843193.abstract
AB Data analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed.