Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

Beyond the intraclass correlation: A hierarchical modeling approach to test-retest assessment

View ORCID ProfileGang Chen, Daniel S. Pine, Melissa A. Brotman, Ashley R. Smith, Robert W. Cox, Simone P. Haller
doi: https://doi.org/10.1101/2021.01.04.425305
Gang Chen
aScientific and Statistical Computing Core, National Institute of Mental Health, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Gang Chen
  • For correspondence: gangchen@mail.nih.gov
Daniel S. Pine
bSection on Development and Affective Neuroscience, National Institute of Mental Health, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Melissa A. Brotman
cNeuroscience and Novel Therapeutics Unit, Emotion and Development Branch, National Institute of Mental Health, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ashley R. Smith
bSection on Development and Affective Neuroscience, National Institute of Mental Health, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Robert W. Cox
aScientific and Statistical Computing Core, National Institute of Mental Health, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Simone P. Haller
cNeuroscience and Novel Therapeutics Unit, Emotion and Development Branch, National Institute of Mental Health, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Preview PDF
Loading

Abstract

The concept of test-retest reliability (TRR) indexes the repeatability or consistency of a measurement across time. High TRR of measures is critical for any scientific study, specifically for the study of individual differences. Evidence of poor TRR of commonly used behavioral and functional neuroimaging tasks is mounting (e.g., Hedge et al., 2018; Elliot et al., 2020). These reports have called into question the adequacy of using even the most common, well-characterized cognitive tasks with robust population-level task effects, to measure individual differences. Here, we demonstrate the limitations of the intraclass correlation coefficient (ICC), the classical metric that captures TRR as a proportional variance ratio. Specifically, the ICC metric is limited when characterizing TRR of cognitive tasks that rely on many individual trials to repeatedly evoke a psychological state or behavior. We first examine when and why conventional ICCs underestimate TRR. Further, based on recent foundational work (Rouder and Haaf, 2019; Haines et al., 2020), we lay out a hierarchical framework that takes into account the data structure down to the trial level and estimates TRR as a correlation divorced from trial-level variability. As part of this process, we examine several modeling issues associated with the conventional ICC formulation and assess how different factors (e.g., trial and subject sample sizes, relative magnitude of cross-trial variability) impact TRR. We reference the tools of TRR and 3dLMEr for the community to apply these models to behavior and neuroimaging data.

Competing Interest Statement

The authors have declared no competing interest.

Copyright 
The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license.
Back to top
PreviousNext
Posted January 05, 2021.
Download PDF
Email

Thank you for your interest in spreading the word about bioRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Beyond the intraclass correlation: A hierarchical modeling approach to test-retest assessment
(Your Name) has forwarded a page to you from bioRxiv
(Your Name) thought you would like to see this page from the bioRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Beyond the intraclass correlation: A hierarchical modeling approach to test-retest assessment
Gang Chen, Daniel S. Pine, Melissa A. Brotman, Ashley R. Smith, Robert W. Cox, Simone P. Haller
bioRxiv 2021.01.04.425305; doi: https://doi.org/10.1101/2021.01.04.425305
Digg logo Reddit logo Twitter logo Facebook logo Google logo LinkedIn logo Mendeley logo
Citation Tools
Beyond the intraclass correlation: A hierarchical modeling approach to test-retest assessment
Gang Chen, Daniel S. Pine, Melissa A. Brotman, Ashley R. Smith, Robert W. Cox, Simone P. Haller
bioRxiv 2021.01.04.425305; doi: https://doi.org/10.1101/2021.01.04.425305

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Neuroscience
Subject Areas
All Articles
  • Animal Behavior and Cognition (3688)
  • Biochemistry (7783)
  • Bioengineering (5673)
  • Bioinformatics (21267)
  • Biophysics (10574)
  • Cancer Biology (8170)
  • Cell Biology (11929)
  • Clinical Trials (138)
  • Developmental Biology (6757)
  • Ecology (10394)
  • Epidemiology (2065)
  • Evolutionary Biology (13853)
  • Genetics (9702)
  • Genomics (13063)
  • Immunology (8136)
  • Microbiology (19976)
  • Molecular Biology (7841)
  • Neuroscience (43032)
  • Paleontology (318)
  • Pathology (1278)
  • Pharmacology and Toxicology (2258)
  • Physiology (3350)
  • Plant Biology (7221)
  • Scientific Communication and Education (1311)
  • Synthetic Biology (2000)
  • Systems Biology (5533)
  • Zoology (1127)