Original ContributionChallenges in diffusion MRI tractography – Lessons learned from international benchmark competitions
Introduction
Diffusion magnetic resonance imaging (dMRI) fiber tractography [1,2] has become a pillar of the neuroimaging community, revealing fundamental insights into how connectivity underlies brain function, development, and cognition, and leading to a better understanding of brain dysfunction in aging, mental health disorders, and neurological disease [3]. Additionally, in the neurosurgical setting, tractography has provided clinically relevant information during pre-operative planning as well as intra-operative mapping of brain tumor resections [4]. Despite its widespread use in both clinical and research domains, the process from data acquisition to generation of 3D connectivity maps is a multi-step procedure with numerous assumptions and uncertainties that can ultimately affect the ability of tractography to faithfully represent the true axonal connections of the brain. Because of this, validating the accuracy and reproducibility of these techniques is critical for sound scientific conclusions and effective surgical outcomes. It is necessary to measure the ability of these techniques to track white matter fibers from region to region, and to also quantify the ability of dMRI to assesses the underlying fiber orientation distribution (FOD) within each voxel. Towards this end, there have been a large number of dMRI community-wide efforts, or “challenges”, which aim to investigate the reliability of the tractography process.
Publicly organized challenges are widespread in biomedical image analysis. In this domain, an algorithm or solution may be developed to address a particular challenge in the field - a challenge that is likely being, or has already been, tackled by multiple laboratories, researchers, and algorithms. However, for many problems, there is no public database and reference standard available, and results are typically reported on proprietary datasets which may vary widely due to differences in acquisition and hardware, making fair comparisons between algorithms not practical. For this reason, public challenges are organized to provide a platform to compare algorithms and results in a fair manner, and evaluate common and emerging algorithms in an effort to advance the state of the field. In biomedical image analysis, challenges have included a number of imaging modalities, anatomies, and evaluation goals. Examples include segmenting tumors and lesions from MR, CT, and PET scans [[5], [6], [7]], detecting pulmonary nodules in chest CT [8], particle tracking [9], evaluating neuronal reconstructions [10,11], estimating knee cartilage [12], motion correction on cardiac data [13], segmenting histological images [14], and segmenting a number of organs [[15], [16], [17], [18]]. A comprehensive list of past and ongoing challenges can be found at https://grand-challenge.org/all_challenges/.
In the tractography community, these challenges provide a unique opportunity for neuroscientists, computer scientists, biomedical engineers, and MRI physicists to fairly compare tractography algorithms in an unbiased format. Traditionally, most have focused on either local modeling of the fiber geometry, or on the evaluation of tractography as a whole. These challenges have provided us with valuable lessons regarding the tractography process –resulting in quantitative measures of the reliability and limitations of existing approaches – and left us with unique opportunities for advancements in brain mapping using ideas and algorithms from different disciplines, research labs, and scientific communities. In this manuscript, we aim to summarize the lessons learned from a decade of challenges in tractography, and to give perspective on the past, present, and future “challenges” that the field of diffusion tractography faces.
We begin with a brief review of “The Challenges” that have provided insight into fiber orientation reconstruction, tractography, and brain connectivity. Next, we summarize the insights and “Lessons Learned” from these studies, and conclude with a discussion on the “Opportunities and Perspectives” on open issues in the field and the future of tractography.
Section snippets
The challenges
The basic anatomy of a challenge includes (1) defining the challenge itself, i.e. the task and desired output, (2) providing a set of images for participants to apply their algorithm on, (3) creating the reference standard, or ground truth, against which all submissions are compared, (4) defining an evaluation procedure, or metrics to quantify performance, and (5) final evaluation of all submissions. In this section, we present a brief history of past tractography challenges in chronological
Lessons learned
These challenges have resulted in a number of insights into the tractography process, its interpretations, strengths, and limitations. We summarize here the main lessons learned from these challenges.
Opportunities and perspectives
We end with a discussion on perspectives on the challenges and the future of tractography, opportunities for advancements, and open issues in the field.
Acknowledgments
This work was supported by the National Institutes of Health under award numbers R01EB017230, and T32EB001628, and by the National Science Foundation under award number 1452485. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or NSF.
References (90)
- et al.
Diffusion MRI at 25: exploring brain tissue structure and function
Neuroimage
(2012) - et al.
White matter tractography for neurosurgical planning: a topography-based review of the current state of the art
Neuroimage Clin
(2017) - et al.
The first MICCAI challenge on PET tumor segmentation
Med Image Anal
(2018) - et al.
Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: the ANODE09 study
Med Image Anal
(2010) - et al.
BigNeuron: large-scale 3D neuron reconstruction from optical microscopy images
Neuron
(2015) - et al.
Benchmarking framework for myocardial tracking and deformation algorithms: an open access database
Med Image Anal
(2013) - et al.
Evaluation of automatic neonatal brain segmentation algorithms: the NeoBrainS12 challenge
Med Image Anal
(2015) - et al.
Sparse Reconstruction Challenge for diffusion MRI: validation on a physical phantom to determine which acquisition scheme and analysis method to use?
Med Image Anal
(2015) - et al.
Quantitative evaluation of 10 tractography algorithms on a realistic diffusion MR phantom
Neuroimage
(2011) - et al.
The role of diffusion tensor imaging in brain tumor surgery: a review of the literature
Clin Neurol Neurosurg
(2014)