Mindcontrol: A Web Application for Brain Segmentation Quality Control
=====================================================================

* Anisha Keshavan
* Esha Datta
* Ian McDonough
* Christopher R. Madan
* Kesshi Jordan
* Roland G. Henry

## Abstract

Tissue classification plays a crucial role in the investigation of normal neural development, brain-behavior relationships, and the disease mechanisms of many psychiatric and neurological illnesses. Ensuring the accuracy of tissue classification is important for quality research and, in particular, the translation of imaging biomarkers to clinical practice. Assessment with the human eye is vital to correct various errors inherent to all currently available segmentation algorithms. Manual quality assurance becomes methodologically difficult at a large scale - a problem of increasing importance as the number of data sets is on the rise. To make this process more efficient, we have developed Mindcontrol, an open-source web application for the collaborative quality control of neuroimaging processing outputs. The Mindcontrol platform consists of a dashboard to organize data, descriptive visualizations to explore the data, an imaging viewer, and an in-browser annotation and editing toolbox for data curation and quality control. Mindcontrol is flexible and can be configured for the outputs of any software package in any data organization structure. Example configurations for three large, open-source datasets are presented: the 1000 Functional Connectomes Project (FCP), the Consortium for Reliability and Reproducibility (CoRR), and the Autism Brain Imaging Data Exchange (ABIDE) Collection. These demo applications link descriptive quality control metrics, regional brain volumes, and thickness scalars to a 3D imaging viewer and editing module, resulting in an easy-to-implement quality control protocol that can be scaled for any size and complexity of study.

## 1. Background

Imaging biomarkers derived from MRI play a crucial role in the fields of neuroscience, neurology, and psychiatry. Estimates of regional brain volumes and shape features can track the disease progression of neurological and psychiatric diseases such as Alzheimer’s disease (Dickerson et al. 2009) (Vemuri and Jack 2010), Parkinson’s disease (Mangia 2013), schizophrenia (Shenton et al. 2001), depression (Meisenzahl et al. 2011), autism (Brambilla et al. 2003), and multiple sclerosis (Filippi et al. 1995). Given recent increases in data collection to accommodate modern precision-medicine approaches, assuring the quality of these biomarkers is vital as we scale their production.

Various semi-automated programs have been developed to estimate MRI biomarkers. While these applications are efficient, errors in regional segmentation are inevitable, given several methodological challenges inherent to both technological and clinical implementation limitations. First, the quality of the MRI scan itself due to motion artifacts or scanner instabilities could blur and distort anatomical boundaries (Blumenthal et al. 2002) (Pardoe et al. 2016) (Reuter et al. 2015) (Savalia et al. 2016). Differences in MRI hardware, software, and acquisition sequences also contribute to contrast differences and gradient distortions that affect tissue classification, which makes combining datasets across sites challenging (Keshavan et al. 2016). An additional source of error comes from parameter selection for segmentation algorithms; different parameter choices can translate to widely varying results (Han et al. 2006). Furthermore, many MR segmentation algorithms were developed and tested on healthy adult brains; applying these algorithms to brain images of children, the elderly, or those with pathology may violate certain assumptions of the algorithm, resulting in drastically different results.

Several quality assurance strategies exist to address segmentation errors. In one approach, researchers flag low-quality scans prior to analysis by viewing the data before input to tissue classification algorithms. However, identifying “bad” datasets using the raw data is not always straightforward, and can be prohibitively time consuming for large datasets. Pre-processing protocols have been developed to extract metrics that can be viewed as a cohort-level summary from which outliers are selected for manual quality-assurance. For example, by running the Preprocessed-Connectomes Project’s Quality Assurance Protocol (PCP-QAP) (Shehzad et al. 2015), researchers can view summary statistics that describe the quality of the raw data going into the algorithm and automatically remove subpar images. However, these metrics are limited because segmentation may still fail even if the quality of the scan is good. Another quality assurance strategy is to plot distributions of the segmentation output metrics themselves and remove any outlier volumes. However, without manual inspection, normal brains that naturally have very small or large estimates of brain size or pathological brains with valid segmentations may be inappropriately removed. Ideally, a link would exist between scalar summary statistics and 3D/4D volumes. Such a link would enable researchers to prioritize images for labor-intensive quality control (QC) procedures; to collaborate and organize QC procedures; and to understand how scalar quality metrics, such as signal to noise ratio, relate to the actual image and segmentation. In this report, we present a collaborative and efficient MRI QC platform that links group-level descriptive statistics to individual volume views of MRI images.

We propose an open source web-based brain quality control application called Mindcontrol: a dashboard to organize, QC, annotate, edit, and collaborate on neuroimaging processing. Mindcontrol provides an intuitive interface for examining distributions of descriptive measures from neuroimaging pipelines (e.g., surface area of right insula), and viewing the results of segmentation analyses using the Papaya.js volume viewer ([https://github.com/riimango/Papaya](https://github.com/riimango/Papaya)). Users are able to annotate points and curves on the volume, edit voxels, and assign tasks to other users (e.g., to manually correct the segmentation of a particular image). The platform is pipeline agnostic, meaning that it can be configured to quality control any set of 3D volumes regardless of what neuroimaging software package produced it. In the following sections, we describe the implementation details of Mindcontrol, as well as its configuration for three open-source datasets, each with a different type of neuroimaging pipeline output.

## 2. Materials and Methods

Mindcontrol was developed with several design constraints. Platform independence is essential so that QC can be performed on any computer or device. Most tablets have limited storage capacity, so space-minimizing specifications were established. Cloud-based data storage was specified to accommodate large neuroimaging datasets and, to efficiently store annotations and voxel editing, Mindcontrol only records the changes to files, rather than whole-file information. Researchers must be able to QC outputs from any type of neuroimaging software package, so Mindcontrol was specified to flexibly accommodate any file organization structure, with configurable “modules” that can contain any type of descriptive statistics and 3D images. Mindcontrol configuration and database updates must require minimal Javascript knowledge, since Matlab/Octave, Python, R, and C are primarily used in the neuroimaging community for data analysis. Finally, changes to the database(like the addition of new images), changes in descriptive measures, and new edits/annotations, should be reflected in the application in real-time to foster collaboration.

### 2.1. Implementation Details

Mindcontrol is built with Meteor ([http://www.meteor.com](http://www.meteor.com)), a full-stack javascript web-development platform. Meteor features a build tool, a package manager, the convenience of a single language (javascript) to develop both the front- and back-end of the application, and an abstracted implementation of full-stack reactivity. Data is transferred “over the wire” and rendered by the client (as opposed to the server sending HTML), which means that changes to the database automatically trigger changes to the application view. For example, as soon as a user finishes implementing QC procedures on an image and clicks “save”, all other users can see the changes. A diagram of this process is provided in Figure 1.

![Figure 1:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2016/11/30/090431/F1.medium.gif)

[Figure 1:](http://biorxiv.org/content/early/2016/11/30/090431/F1)

Figure 1: 
This diagram shows the different components of the Mindcontrol application. A) The client sends information, such as annotations and edits, to the server. B) The server calls a method that updates the mongoDB backend. C) When the back-end MongoDB database changes, these changes are automatically pushed to the minimongo database on the client. D) Change to the minimongo database automatically re-renders the view on the client. E) Users can optionally push changes to the client view via the MongoDB with Python MongoDB drivers. Drivers for C, C++, Scala, Java, PHP, Ruby, Perl, and Node.js are also available through MongoDB. F) Developers can optionally write server methods to launch Python or command-line processes that, in turn, use user annotations and edits to re-process images and update the MongoDB with new results.

The user interface consists of a dashboard view and an imaging view, as shown in Figures 2 and 3, respectively. The primary dashboard view consists of processing module sections, a query controller, data tables, and descriptive statistic visualizations. The processing modules are defined by the user in the JSON structure, which describes the module names and columns to display in the data table, below. Each entry in the table is a link that, when clicked, filters all tables on the page. The filters or queries can be saved, edited, and loaded in the query controller section, as shown in Figure 5.

![Figure 2:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2016/11/30/090431/F2.medium.gif)

[Figure 2:](http://biorxiv.org/content/early/2016/11/30/090431/F2)

Figure 2: 
This figure shows the Mindcontrol layout configured to quality check Freesurfer outputs from the 1000 Functional Connectomes Project (FCP). Part A shows the module navigator, which links to the different processing modules on the dashboard. Part B shows the different exams and the dates they were acquired as a heatmap, where green is more and orange is less scans collected on a given day. (For demonstration purposes, the dates depicted here do not reflect the actual dates the data were collected for the FCP, since this information was not provided at the time.) Clicking on data in any column of the exam table filters the data by that column. For example, clicking the site “Milwaukee” reduces both the “Exams” and the “FreeSurfer” tables to only show subjects from Milwaukee. Part C shows the Freesurfer table and regional volume distribution of the left caudate. A drop-down menu allows users to switch the descriptive metric. Clicking on a value in FreeSurfer ID column brings the user to the imaging view, as shown in Figure 3, where users can evaluate and annotate the quality status of the image. The value of the label in the “QC” column changes instantaneously due to Meteor’s built in full-stack reactivity.

![Figure 3:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2016/11/30/090431/F3.medium.gif)

[Figure 3:](http://biorxiv.org/content/early/2016/11/30/090431/F3)

Figure 3: 
The imaging view of mindcontrol consists of a panel on the left-hand side that contains the QC status; a point annotation menu; a curve annotation menu; a voxel editing menu; and an editable sub-panel for QC status, notes, and editor assignment. On the right-hand side, the base MRI anatomical MPRAGE image is displayed with an overlay of the Freesurfer segmentation outputs using the Papaya.js viewer.

![Figure 4:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2016/11/30/090431/F4.medium.gif)

[Figure 4:](http://biorxiv.org/content/early/2016/11/30/090431/F4)

Figure 4: 
This demonstrates the interactive brushing feature of Mindcontrol histograms. On the left, the user has brushed the tail end of the left caudate volume distribution from Freesurfer. On the right, the histogram has been redrawn with data from the brushed range, and the table beneath filtered from 200 entries to 14 entries based on the brushed caudate volumes.

![Figure 5:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2016/11/30/090431/F5.medium.gif)

[Figure 5:](http://biorxiv.org/content/early/2016/11/30/090431/F5)

Figure 5: 
The query controller shows the different filters that have been applied to this dataset. In this example, the exams have been filtered by institution (“Milwaukee”) and by a range of left caudate volumes (brushed from the histogram). Clicking the “x” next to the filter removes it, and the view updates. Queries can be saved and reloaded by providing a name in the text-entry box. “Reset” removes all filters to show the whole dataset.

Descriptive statistics are visualized using the D3 library ([https://d3js.org/](https://d3js.org)). Currently, two visualization are provided: a histogram of calendar dates that shows the number of exams collected on a given day and 1D histograms with dimensions that are swappable using a dropdown menu, as shown in Figure 2. Both histogram plots interactively filter the data tables below. Clicking on a particular date on the date-histogram plot filters all tables by the exams collected on that particular date. Users are able to “brush” sections of the histogram to filter all tables with exams that meet requirements of values within that range.

The imaging view is shown in Figure 3. The left-side column includes a section to mark an image as “Pass”, “Fail”, “Edited”, or “Needs Edits” and to provide notes. The status bar at the top-left portion updates instantaneously with information on which user checked the image, the quality status of the image, and when it was last checked. Users are also able to assign edits to be performed by other users on the system; for example, a research assistant can perform a general QC and assign difficult cases to a neuroradiologist. On the right-hand side, the Papaya.js viewer ([http://rii-mango.github.io/Papaya/](http://rii-mango.github.io/Papaya)) is used to display the NifTI volumes of the original data and FreeSurfer segmentations. Images must be hosted on a separate server or a content delivery network (CDN) and the Mindcontrol database populated with URLs to these images.

Annotations of points and curves are shown in Figure 6. Using the *shift* key, users can click on the image to annotate points or select the “Logged Curves” toolbar. By *shift+click and dragging*, users can draw curves. Keyboard and mouse shortcuts provided by the Papaya.js viewer, along with Mindcontrol, include toggling overlays (*zz*) and undoing annotations (*dd*). Figure 7 shows the editing (“Painter”) panel of the imaging view. Users set paintbrush values and *shift+click and drag* to change voxel values. For point and curve annotations and voxel editing, the images themselves are not changed, but world x,y,z coordinates, along with annotation text or paintbrush values, are saved to the mongo database when the user clicks “save”. Custom offline functions may be written to apply editing to images: for example, to implement pial surface edits from FreeSurfer.

![Figure 6:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2016/11/30/090431/F6.medium.gif)

[Figure 6:](http://biorxiv.org/content/early/2016/11/30/090431/F6)

Figure 6: 
The annotations panel can be used to annotate a single point (shown in red, part A) and curves (shown in B). When annotating points, the user is shown the selected x,y,z world coordinates and is able to name the annotation. In the curve annotation panel on the left sidebar, the user is able to name the curve and add/remove curves. Keyboard shortcuts: “dd” removes the previous annotation and “zz” toggles the segmentation overlay.

![Figure 7:](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2016/11/30/090431/F7.medium.gif)

[Figure 7:](http://biorxiv.org/content/early/2016/11/30/090431/F7)

Figure 7: 
The editing panel on the left shows the “Painter” toolbox in yellow, where users can input brush values or use the eyedropper tool to set the value to that of a clicked label. The eraser icon sets the brush value to 0, to delete or erase voxels. In the image above, the Freesurfer segmentation is being edited by erasing the voxels missclassified as dura.

### 2.2. Applications

Mindcontrol configurations were developed for selected data from the 1000 Functional Connectomes project (FCP), the consortium for reliability and reproducibility (CoRR), and the Autism Brain Imaging Data Exchange (ABIDE) Collection I.

The FCP consists of 1414 resting state fMRI and corresponding structural datasets collected from 35 sites around the world (Biswal et al. 2010), which have been openly shared with the public. The purpose of the FCP collaboration is to comprehensively map the functional human connectome, to understand genetic influences on brain’s structure and function, and to understand how brain structure and function relate to human behavior (Biswal et al. 2010). Segmentation of 200 selected FCP anatomical images from Baltimore, Bangor, Berlin, ICBM, and Milwaukee was performed with Freesurfer (recon-all) version 5.3.0 (Fischl et al. 2002) using the RedHat 7 operating system on IEEE 754 compliant hardware. Regional volumes of subcortical and cerebellar regions were computed. Cortical volumes, areas, and thicknesses were also computed and averaged across hemispheres. Scan dates were simulated in order to demonstrate the date histogram shown in Figure 1B. The original anonymized T1-weighted images, along with the aparc+aseg output from Freesurfer, were converted to the compressed NifTi (.nii.gz) format and uploaded to Dropbox for the purpose of visualization within Mindcontrol. The Mindcontrol database was populated with URLs to these images, along with their corresponding FreeSurfer segmentation metrics.

The purpose of CoRR is to provide an open-science dataset to assess the reliability of functional and structural connectomics by defining test-retest reliability of commonly used MR metrics; to understand the variability of these metrics across sites; and to establish a standard benchmark dataset on which to evaluate new imaging metrics (Zuo et al. 2014). The Preprocessed-Connectomes Project’s Quality Assurance Protocol (PCP-QAP) software was developed to provide anatomical and functional data quality measures in order to detect low-quality images before data processing and analysis (Shehzad et al. 2015). PCP-QAP normative data for the CoRR study was downloaded from [https://github.com/preprocessed-connectomes-project/quality-assessment-protocol](https://github.com/preprocessed-connectomes-project/quality-assessment-protocol). The Mindcontrol database was populated with pointers to 2,963 CoRR structural images residing on an Amazon S3 bucket along with their corresponding PCP-QAP metrics. Some metrics include contrast-to-noise ratio, signal-to-noise ratio, voxel smoothness, percentage of artifact voxels, foreground-to-background energy ratio, and entropy focus criterion (Shehzad et al. 2015).

The overarching goal of the ABIDE initiative is to expedite the discovery of the neural basis of autism by providing open access to a large, heterogeneous collection of structural and functional neuroimaging datasets collected from over 23 institutions (Martino et al. 2013). The preprocessed connectomes project provides cortical thickness measures of the ABIDE dataset output by the ANTs software package (Avants et al. 2009), along with summary statistics across regions of interests (ROIs) defined by the Desikan-Killiany-Tourville (DKT) protocol (Klein and Tourville 2012). The Mindcontrol database was populated with pointer URLs to S3-hosted cortical thickness images and their corresponding ROI summary measures, along with PCP-QAP metrics. The FCP, CoRR, and ABIDE Mindcontrol configurations are currently hosted on Heroku ([https://www.heroku.com/](https://www.heroku.com)).

## 3. Implementation

The demo of the FCP data is located at [http://mindcontrol.herokuapp.com](http://mindcontrol.herokuapp.com). The demo of the CoRR dataset with PCP-QAP metrics is hosted at [http://mindcontrol-corr.herokuapp.com](http://mindcontrol-corr.herokuapp.com). The demo of the ABIDE dataset with ANTS cortical thickness metrics is located at [http://mindcontrol-abide.herokuapp.com](http://mindcontrol-abide.herokuapp.com). The Mindcontrol codebase is hosted on GitHub at [http://github.com/akeshavan/mindcontrol](http://github.com/akeshavan/mindcontrol), along with installation instructions. The Mindcontrol configuration of the FCP data is located on the master branch of the GitHub repository, and the configurations for CoRR and ABIDE are located at [http://github.com/akeshavan/mindcontrol_configs](http://github.com/akeshavan/mindcontrol_configs), along with configuration documentation.

## 4. Discussion

Mindcontrol is a configurable neuroinformatics dashboard that links study information and descriptive statistics with scientific data visualization, MR images, and their overlays (segmentation or otherwise). The three configurations demonstrated in this report show the link between MRI quality metrics and raw data, the link between Freesurfer regional volumes and segmentation quality, and the link between ANTS cortical thickness summary statistics and segmentation/thickness estimates of the volume. The platform is configurable, open-source, and software/pipeline agnostic, enabling researchers to configure it to their particular analyses. The dashboard allows researchers to assign editing tasks to others, who can then perform edits on the application itself.

There have been considerable efforts in this field to ensure data quality on a large scale. The human connectome project’s extensive informatics pipeline, which includes a database service, QC procedures, and a data visualization platform, has been key to the project’s success in collecting a large, high-quality dataset (Marcus et al. 2013). The Allen Brain Atlas offers a comprehensive genetic, neuroanatomical, and connectivity web-based data exploration portal, linking an MRI viewer to data tables (Sunkin et al. 2012). The open-source LORIS web-based data management system integrates an image viewer with extensive QC modules (Das et al. 2012). Mindcontrol supplements these efforts by providing a lightweight and extensible data management and visualization system with the added capability to perform edits and curate annotations within the application.

Mindcontrol is actively being used at UCSF to study imaging biomarkers of multiple sclerosis (MS) disease progression in a 12-year longitudinal cohort of over 500 patients. MRI plays a crucial role in the diagnosis of MS due to its sensitivity to the white matter lesions characteristic of this disease (Ge 2006). Both the location and number of lesions are used in the diagnostic criteria of MS (McDonald et al. 2001); their evolution over time is considered a proxy for disease progression, and is closely monitored as a key outcome measure in clinical trials (Ge et al. 2000). Researchers have identified numerous imaging biomarkers of disease progression, including cortical atrophy (Fisher et al. 2008) and sensitive lesion quantification methods from FLAIR sequences (Schmidt et al. 2012).

Currently, Mindcontrol is being used as a data management dashboard and plays a role in the semi-automated identification of multiple sclerosis lesions. The Lesion Segmentation Toolbox (LST) automates the detection of FLAIR hyper-intense lesions, which prevents user-bias of the lesion boundary (Schmidt et al. 2012). However, because our acquisition protocols sufficiently differ from those of the LST developers, manual intervention by neuroradiologists is often necessary to correct false positives and false negatives in our dataset. Mindcontrol’s point annotation feature is used by neuroradiologists to mark the center of a false positive lesion. The curve annotation feature is used to quickly note the rough outline of a false negative lesion. Custom Python scripts use these annotations to produce a corrected lesion segmentation map, which is then used to tune the parameters of the LST. This method avoids biasing the lesion boundary and saves time by allowing users to quickly mark lesions with one click instead of the traditional method of coloring the full lesion volume by hand. Distributions of the lesion count and lesion volume are used by the researchers to prioritize the cases requiring more intensive editing.

Mindcontrol is also being used at UCSF to coordinate a large-scale brain mask editing effort of all >4000 exams in the MS cohort. Using Mindcontrol’s voxel editing or “painting” feature on a Microsoft Surface Pro 4 tablet with stylus, multiple research assistants are collaboratively erasing voxels incorrectly marked as brain and painting brain voxels that were not identified. These edits are done with a stylus, which feels more natural and is faster than using a mouse. Researchers assign more complicated edits to the experts in the lab, write questions and comments in the “notes” section, and mark the brain as “Edited” once it is complete. A nightly cron-job applies their edits to the brain masks and assigns a reviewer to perform a final QC. All brains marked with a “Pass” are added to a set of brain masks to be incorporated into a training set for the mincbeast brain extraction algorithm (Eskildsen et al. 2012). Leveraging Mindcontrol as a semi-automated segmentation utility has made our processing methods more efficient, organized, precise, and collaborative.

## 5. Future Directions

Mindcontrol is being actively developed to incorporate new features that will improve outlier detection, efficiency, and collaboration. New information visualizations to detect outliers include: scatter plots to compare two metrics against each other, and a longitudinal view of a single-subject trajectory for a given metric to detect uncharacteristic temporal changes. New scientific data visualizations are planned using the BrainBrowser library (Sherif et al. 2015) to display cortical surfaces. A beta version of real-time collaborative annotations, where two users can annotate the same image and see the edits of the other user as they occur, is in the testing phase.

Currently, configuring Mindcontrol involves creating one JSON file to describe the different modules and another JSON file to populate the Mongo database with pointers to images and their scalar metrics. In the future, this process could be streamlined by creating a Mindcontrol configuration for datasets with a standardized folder structure, like the Brain Imaging Data Structure (BIDS) (Gorgolewski et al. 2016), and their BIDS-compliant derivatives (Gorgolewski et al. 2016). Further development of Mindcontrol will include the flexible importing of additional scalar metrics, such as measures of structural complexity, calculated by third-party toolboxes developed to complement standard analysis pipelines (Madan and Kensinger 2016) (Madan and Kensinger 2017). This will enable researchers to collaborate on the same dataset by uploading metrics from their newly developed algorithms, and will enable them to easily explore their results in the context of metrics contributed by others. Finally, Mindcontrol has the potential to be a large-scale crowd-sourcing platform for segmentation editing and quality control. We hope the functionality, ease-of-use, and modularity offered by Mindcontrol will help to improve the standards used by studies relying on brain segmentation. We are moving into a world of big data, with an unprecedented amount of information available on each individual. This explosion of information availability has the capacity to revolutionize research and treatment of complex diseases. To harness the potential of these big data initiatives, platforms such as Mindcontrol is crucial as we move forward into this new phase of interdisciplinary translational research.

## 6. Acknowledgements

AK would like to acknowledge Satrajit Ghosh and Arno Klein for their mentorship, the Papaya.js developers for their support, and Cameron Craddock for his assistance with the CoRR and ABIDE datasets. AK, ED, IM, KJ, and CM would like to thank the organizers of Neurohackweek and the Brainhack events for fostering a collaborative environment in which to develop new ideas and software. CM was supported by a fellowship from the Canadian Institutes of Health Research (FRN-146793). ED and KJ were supported by the National Defense Science and Engineering Graduate Fellowship. Icons from Figure 1 were sourced from [https://icons8.com](https://icons8.com).

*   Received November 29, 2016.
*   Accepted November 30, 2016.


*   © 2016, Posted by Cold Spring Harbor Laboratory

This pre-print is available under a Creative Commons License (Attribution-NonCommercial 4.0 International), CC BY-NC 4.0, as described at [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)

## References

1.  Dickerson BC, Bakkour A, Salat DH, and others (2009) The cortical signature of Alzheimer’s disease: regionally specific cortical thinning relates to symptom severity in very mild to mild AD dementia and is detectable in asymptomatic amyloid-positive individuals. Cereb Cortex 19:497–510.
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1093/cercor/bhn113&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=18632739&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000263162600001&link_type=ISI) 

2.  Vemuri P., Jack CR (2010) Role of structural MRI in Alzheimer's disease. Alzheimers Res Ther 2:23. doi: 10.1186/alzrt47
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1186/alzrt47&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=20807454&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 

3.  Mangia PJTS (2013) Magnetic Resonance Imaging (MRI) in Parkinson’s Disease. Journal of Alzheimer’s Disease & Parkinsonism. doi: 10.4172/2161-0460.s1-001
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.4172/2161-0460.s1-001&link_type=DOI) 

4.  Shenton ME, Dickey CC, Frumin M, McCarley RW (2001) A review of MRI findings in schizophrenia. Schizophrenia research 49:1–52.
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/S0920-9964(01)00163-3&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=11343862&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000169189900001&link_type=ISI) 

5.  Meisenzahl E, Koutsouleris N, Reiser M, and others (2011) Structural MRI correlates for vulnerability and resilience to major depressive disorder. Journal of psychiatry & neuroscience: JPN 36:15.
    
    

6.  Brambilla P, Hardan A, di Nemi SU, and others (2003) Brain anatomy and development in autism: review of structural MRI studies. Brain research bulletin 61:557–569.
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.brainresbull.2003.06.001&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=14519452&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000189133800001&link_type=ISI) 

7.  Filippi M, Horsfield MA, Tofts PS, and others (1995) Quantitative assessment of MRI lesion load in monitoring the evolution of multiple sclerosis. Brain 118:1601–1612. doi:10.1093/brain/118.6.1601
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1093/brain/118.6.1601&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=8595489&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=A1995TQ33300020&link_type=ISI) 

8.  Blumenthal JD, Zijdenbos A, Molloy E, Giedd JN (2002) Motion Artifact in Magnetic Resonance Imaging: Implications for Automated Analysis. NeuroImage 16:89–92. doi:10.1006/nimg.2002.1076
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1006/nimg.2002.1076&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=11969320&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000176624700009&link_type=ISI) 

9.  Pardoe HR, Hiess RK, Kuzniecky R (2016) Motion and morphometry in clinical and nonclinical populations. NeuroImage 135:177–185. doi: 10.1016/j.neuroimage.2016.05.005
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2016.05.005&link_type=DOI) 

10. Reuter M, Tisdall MD, Qureshi A, and others (2015) Head motion during MRI acquisition reduces gray matter volume and thickness estimates. NeuroImage 107:107–115. doi:10.1016/j.neuroimage.2014.12.006
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2014.12.006&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=25498430&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 

11. Savalia NK, Agres PF, Chan MY, and others (2016) Motion-related artifacts in structural brain images revealed with independent estimates of in-scanner head motion. Human Brain Mapping. doi: 10.1002/hbm.23397
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1002/hbm.23397&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=27634551&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 

12. Keshavan A, Paul F, Beyer MK, and others (2016) Power estimation for non-standardized multisite studies. NeuroImage 134:281–294.
    
    

13. Han X, Jovicich J, Salat D, and others (2006) Reliability of MRI-derived measurements of human cerebral cortical thickness: The effects of field strength scanner upgrade and manufacturer. NeuroImage 32:180–194. doi: 10.1016/j.neuroimage.2006.02.051
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2006.02.051&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=16651008&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000239530700020&link_type=ISI) 

14. Shehzad Z, Giavasis S, Li Q, and others (2015) The Preprocessed Connectomes Project Quality Assessment Protocol: A resource for measuring the quality of MRI data. Frontiers in Neuroscience. doi: 10.3389/conf.fnins.2015.91.00047
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.3389/conf.fnins.2015.91.00047&link_type=DOI) 

15. Giedd JN, Rapoport JL (2010) Structural MRI of pediatric brain development: what have we learned and where are we going?. Neuron 67:728–734.
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuron.2010.08.040&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=20826305&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000281938900008&link_type=ISI) 

16. Biswal BB, Mennes M, Zuo X-N, and others (2010) Toward discovery science of human brain function. Proceedings of the National Academy of Sciences 107:4734–4739.
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMToiMTA3LzEwLzQ3MzQiO3M6NDoiYXRvbSI7czozNzoiL2Jpb3J4aXYvZWFybHkvMjAxNi8xMS8zMC8wOTA0MzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

17. Fischl B, Salat DH, Busa E, and others (2002) Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron 33:341–355.
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/S0896-6273(02)00569-X&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=11832223&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000173643200006&link_type=ISI) 

18. Zuo X-N, Anderson JS, Bellec P, and others (2014) An open science resource for establishing reliability and reproducibility in functional connectomics. Scientific Data 1:140049. doi: 10.1038/sdata.2014.49
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/sdata.2014.49&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=25977800&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 

19. Martino AD, Yan C-G, Li Q, and others (2013) The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Molecular Psychiatry 19:659–667. doi: 10.1038/mp.2013.78
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/mp.2013.78&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=23774715&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 

20. Avants BB, Tustison N, Song G (2009) Advanced normalization tools (ANTS). Insight J 2:1–35.
    
    

21. Klein A, Tourville J (2012) 101 Labeled Brain Images and a Consistent Human Cortical Labeling Protocol. Front Neurosci. doi: 10.3389/fnins.2012.00171
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.3389/fnins.2012.00171&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=23227001&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 

22. Marcus DS, Harms MP, Snyder AZ, and others (2013) Human Connectome Project informatics: Quality control database services, and data visualization. NeuroImage 80:202–219. doi: 10.1016/j.neuroimage.2013.05.077
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2013.05.077&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=23707591&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 

23. Sunkin SM, Ng L, Lau C, and others (2012) Allen Brain Atlas: an integrated spatio-temporal portal for exploring the central nervous system. Nucleic Acids Research 41:D996–D1008. doi: 10.1093/nar/gks1042
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1093/nar/gks1042&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=23193282&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000312893300142&link_type=ISI) 

24. Das S, Zijdenbos AP, Harlap J, and others (2012) LORIS: a web-based data management system for multi-center studies. Front Neuroinform. doi: 10.3389/fninf.2011.00037
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.3389/fninf.2011.00037&link_type=DOI) 

25. Ge Y (2006) Multiple sclerosis: the role of MR imaging. American Journal of Neuroradiology 27:1165–1176.
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo5OiIyNy82LzExNjUiO3M6NDoiYXRvbSI7czozNzoiL2Jpb3J4aXYvZWFybHkvMjAxNi8xMS8zMC8wOTA0MzEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

26. McDonald WI, Compston A, Edan G, and others (2001) Recommended diagnostic criteria for multiple sclerosis: guidelines from the International Panel on the diagnosis of multiple sclerosis. Annals of neurology 50:121–127.
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1002/ana.1032&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=11456302&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000169561300017&link_type=ISI) 

27. Ge Y, Grossman RI, Udupa JK, and others (2000) Glatiramer acetate (Copaxone) treatment in relapsing–remitting MS Quantitative MR assessment. Neurology 54:813–817.
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1212/WNL.54.4.813&link_type=DOI) 

28. Fisher E, Lee J-C, Nakamura K, Rudick RA (2008) Gray matter atrophy in multiple sclerosis: a longitudinal study. Annals of neurology 64:255–265.
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1002/ana.21436&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=18661561&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000259681700008&link_type=ISI) 

29. Schmidt P, Gaser C, Arsic M, and others (2012) An automated tool for detection of FLAIR-hyperintense white-matter lesions in Multiple Sclerosis. NeuroImage 59:3774–3783. doi: 10.1016/j.neuroimage.2011.11.032
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2011.11.032&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=22119648&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 

30. Eskildsen SF, Coupé P, Fonov V, and others (2012) BEaST: brain extraction based on nonlocal segmentation technique. NeuroImage 59:2362–2373.
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2011.09.012&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=21945694&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 

31. Sherif T, Kassis N, Rousseau M-
    
    

32. Gorgolewski KJ, Auer T, Calhoun VD, and others (2016) The brain imaging data structure a format for organizing and describing outputs of neuroimaging experiments. Scientific Data 3:160044. doi: 10.1038/sdata.2016.44
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/sdata.2016.44&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=27326542&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 

33. Gorgolewski KJ, Alfaro-Almagro F, Auer T, and others (2016) BIDS Apps: Improving ease of use, accessibility and reproducibility of neuroimaging data analysis methods. bioRxiv 079145.
    
    

34. Madan CR, Kensinger EA (2016) Cortical complexity as a measure of age-related brain atrophy. NeuroImage 134:617–629. doi: 10.1016/j.neuroimage.2016.04.029
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2016.04.029&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=27103141&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2016%2F11%2F30%2F090431.atom) 

35. Madan CR, Kensinger EA (2017) Age-related differences in the structural complexity of subcortical and ventricular structures. Neurobiology of Aging. doi: 10.1016/j.neurobiolaging.2016.10.023
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neurobiolaging.2016.10.023&link_type=DOI)