Quantifying the reliability of image replication studies: the image intraclass correlation coefficient (I2C2)

Cogn Affect Behav Neurosci. 2013 Dec;13(4):714-24. doi: 10.3758/s13415-013-0196-0.

Abstract

This article proposes the image intraclass correlation (I2C2) coefficient as a global measure of reliability for imaging studies. The I2C2 generalizes the classic intraclass correlation (ICC) coefficient to the case when the data of interest are images, thereby providing a measure that is both intuitive and convenient. Drawing a connection with classical measurement error models for replication experiments, the I2C2 can be computed quickly, even in high-dimensional imaging studies. A nonparametric bootstrap procedure is introduced to quantify the variability of the I2C2 estimator. Furthermore, a Monte Carlo permutation is utilized to test reproducibility versus a zero I2C2, representing complete lack of reproducibility. Methodologies are applied to three replication studies arising from different brain imaging modalities and settings: regional analysis of volumes in normalized space imaging for characterizing brain morphology, seed-voxel brain activation maps based on resting-state functional magnetic resonance imaging (fMRI), and fractional anisotropy in an area surrounding the corpus callosum via diffusion tensor imaging. Notably, resting-state fMRI brain activation maps are found to have low reliability, ranging from .2 to .4. Software and data are available to provide easy access to the proposed methods.

Publication types

  • Research Support, N.I.H., Extramural

MeSH terms

  • Adult
  • Brain / anatomy & histology
  • Brain / pathology
  • Brain / physiology*
  • Brain Mapping*
  • Computer Simulation
  • Female
  • Humans
  • Male
  • Models, Biological
  • Neuroimaging* / classification
  • Reproducibility of Results
  • Statistics as Topic*