Automatic recognition of cortical sulci of the human brain using a congregation of neural networks
Introduction
The development of image analysis methods dedicated to automatic management of brain anatomy is a widely addressed area of research. A number of works focus on the notion of deformable atlases, which can be elastically transformed to reflect the anatomy of new subjects. An exhaustive bibliography of this approach initially proposed by Bajcsy and Broit (1982) is largely beyond the scope of this paper (see (Thompson et al., 2000) for a recent review). The complexity and the striking inter-individual variability of the human cortex folding patterns, however, have led several groups to question the behaviour of the deformable atlas framework at the cortex level (Mangin et al., 1995b, Collins et al., 1998, Hellier and Barillot, 2002, Lohmann and von Cramon, 2000, Cachier et al., 2001). Two main issues have to be addressed:
- 1.
What are the features of the cortex folding patterns which should be matched across individuals? While some sulci clearly belong to this set of landmark features because they are usually considered as boundaries between different functional areas, nobody knows to which extent secondary folds should play the same role (Welker, 1989, Régis et al., 1995). Some answers to this important issue could stem from foreseeable advances in mapping brain functional organization (Watson et al., 1993) and connectivity (Poupon et al.). While the number of reliable landmarks to be matched is today relatively limited, comparison of deformable atlas methods at the cortex level should focus on the pairing of these landmarks.
- 2.
Deformable atlas methods rely on the optimization of some function which realizes a trade-off between similarity to the new brain and deformation cost. Whatever the approach, the function driving the deformations is non-convex. When high-dimensional deformation fields are used, this non-convexity turns out to be particularly problematic since standard optimization approaches are bound to lead to a local optimum. While multi-resolution methods may guarantee that an ‘interesting optimum’ is found, the complexity of the cortical folding patterns implies that a lot of other similar optima exist. An important issue is raised by this observation: is the global optimum the best one according to the pairing of sulcal landmarks? The answer to this issue should be taken into account when comparing various approaches.
The various issues mentioned above have led us to initiate a long term project aiming first at a better understanding of the cortical folding patterns (Mangin et al., 1995a, Régis et al., 1995), and second at the automatic identification of the main sulci (Mangin et al., 1995b). During a feasibility study, this project led to a first generation of image analysis tools extracting automatically each cortical fold from a T1-weighted MR image. Then, a sophisticated browser allowed our neuroanatomist to navigate through various 3D representations of the cortical patterns in order to identify the main sulci. This visualization tool led to the creation of a database of brains in which a name was given to each fold. This database was used to train an automatic sulcus recognition system based on a random graph model. Any cortical folding pattern was considered as a realization of this model, which led us to formalize the recognition process as a consistent labelling problem. The solution was obtained from a maximum a posteriori estimator designed in a Markovian framework. While this first tool generation has been used for four years for the planning of depth electrode implantation in the context of epilepsy surgery (about 40 operations), a number of serious flaws had to be overcome to allow a wider use of the toolbox. This paper gives an overview of the second tool generation with emphasis on the more important improvement, which consists in using standard neural nets to build a better model of the random graph probability distribution.
Our approach may be considered as a symbolic version of the deformable atlas approach. The framework is made up of two stages. An abstract structural representation of the cortical topography is extracted first from each new T1-weighted MR image. This representation is supposed to include all the information required to identify sulci. A contextual pattern recognition method is then used to label automatically cortical folds. This method can be interpreted as a graph matching approach. Hence, the usual iconic anatomical template is replaced by an abstract structural template. The one to many matching between the template nodes and the nodes of one structural representation is simply a labelling operation. This labelling is driven by the minimization of a global function made up of local potentials. Each local potential is a measure of the likelihood of the labelling of a restricted cortex area. This potential is given by a virtual expert in this area made up of a multi-layer perceptron trained on a learning database.
While the complexity of the preprocessing stage required by our method may appear as a weakness compared to the straightforward use of continuous deformations, it results in a fundamental difference. While the evaluation of functions driving continuous deformations is costly in terms of computation, the function used to drive the symbolic recognition relies on only a few hundred labels and can be evaluated at a low cost. Hence, stochastic optimization algorithms can be used to deal with the non-convexity problems. In fact, working at a higher level of representation leads to more efficiency for the pattern recognition process, which explains an increasing interest in the community (Lohmann and von Cramon, 1998, Lohmann and von Cramon, 2000, Le Goualher et al., 1998, Le Goualher et al., 1999).
In the following, the second section summarizes the main steps of the preprocessing stage. The third section gives an overview of the building-up of a database of manually labelled brains used to teach cortical anatomy to the pattern recognition system. The fourth section introduces the probabilistic framework underlying the graph matching procedure. The fifth section focuses on the training of the artificial neural networks. The sixth section describes the stochastic minimization heuristics and some results. Finally, the last section highlights the fact that improving the current system will require collaborative work with various neuroscience teams.
Section snippets
The preprocessing stage
This section describes briefly the robust sequence of treatments that automatically converts a T1-weighted MR image into an abstract structural representation of the cortical topography. The whole sequence requires about half an hour on a conventional workstation. All the steps have been validated with at least 50 different images, some of them with several hundred. These images have been acquired with 6 different scanners using various MR sequence parameters. Several experiments have led us to
The learning database
Our preprocessing tool can be viewed as a compression system which provides for each individual brain a synthetic description of the cortex folding patterns. A sophisticated 3D browser allows our neuroanatomist to label manually each node with a name chosen in a list of anatomical entities. The lack of a validated explanation of the structural variability of the human cortex is an important problem during this labelling. Indeed, standard sulci are often split into several folds with various
The random graph and Markovian models
The structural model underlying our pattern recognition system is a random graph, namely a structural prototype whose vertices and relations are random variables (Fig. 6). In order to allow vertices and relations of the random graph to yield sets of several nodes or several links in individual brains, the classical definition proposed by Wong and You (1985) is extended by substituting the monomorphism by a homomorphism (Mangin et al., 1995b). The recognition process can be formalized as a
MLP topology and pattern coding
The choice of MLP topology (number of layers, number of neurons in each layer, connectivity) is known to be a difficult problem without general solution. For our application where a lot of different MLPs have to be designed, an adaptive strategy may have been the best choice. In the following, however, only two different topologies will be used: one for sulcus experts and one for relationship experts. The small size of our learning database, indeed, prevents a consistent adaptive strategy to be
Results
The 265 expert training process has been performed on a network of ten standard workstations and lasts about 24 h. Of course, while this high training cost was cumbersome during the tuning of the system, it is acceptable in a standard exploitation situation. Indeed, this training is done only one time, or more exactly each time we decide to enlarge the learning database.
Conclusion
A number of approaches relying on the deformable atlas paradigm consider that anatomical a priori knowledge can be completely embedded in iconic templates. While this point of view is very powerful for anatomical structures presenting low inter-individual variability, it seems insufficiently versatile to deal with the human cortical anatomy. This observation has led several teams to investigate approaches relying on higher levels of representation. All these approaches rely on a preprocessing
References (43)
- et al.
Automatic labelling of the human cortical surface using sulcal basins
Medical Image Analysis
(2000) - et al.
Towards inference of human brain connectivity from MR diffusion tensor data
Medical Image Analysis
(2001) - et al.
Detection and statistical analysis of human cortical sulci
NeuroImage
(1999) - et al.
Finding parametric representations of the cortical sulci using active contour model
Medical Image Analysis
(1997) - et al.
Matching of deformed images
- et al.
A mean curvature based primal sketch to study the cortical folding process from antenatal to adult brain
- et al.
Multipatient registration of brain MRI using intensity and geometric features
- et al.
Using local geometry to build 3D sulcal models
- et al.
Registration of cortical anatomical structures via robust 3D point matching
- et al.
Non-linear cerebral registration with sulcal constraints
Automatic retrieval of anatomical structures in 3D medical images
Spatial registration and normalization of images
Hum. Brain Mapping
Stochastic relaxation, Gibbs distributions, and the bayesian restoration of images
IEEE Proc. Am. Med. Inst.
Cooperation between local and global approaches to register brain images
Modeling cortical sulci using active ribbons
Int. J. Pattern Recognit. Artific. Intell.
Automatic identification of cortical sulci using a 3D probabilistic atlas
Automated extraction and variability analysis of sulcal neuroanatomy
IEEE Trans. Med. Imaging
Retrospective correction of MR intensity inhomogeneity by information minimization
Automatic detection and labelling of the human brain cortical folds in MR data sets
Topological segmentation of discrete surfaces
Int. J. Comput. Vis.
Differential features of cortical folds
Cited by (182)
The emergence of multiscale connectomics-based approaches in stroke recovery
2024, Trends in NeurosciencesLabeling lateral prefrontal sulci using spherical data augmentation and context-aware training
2021, NeuroImageCitation Excerpt :These latter studies manually defined hundreds to thousands of tertiary sulci as a majority of present tools do not fully support labeling of fine-grained tertiary sulci in all association cortices yet. Nevertheless, steady improvements of previous tools are bringing us closer to achieving this goal (Borne et al., 2020; Cointepas et al., 2001; Joshi et al., 2012; Le Goualher et al., 1999; Lyu et al., 2010; Mangin et al., 1995; Parvathaneni et al., 2019b; Perrot et al., 2008; Rettmann et al., 2002; Riviere et al., 2002; Sandor and Leahy, 1997; Shattuck et al., 2009; Shi et al., 2009; Tao et al., 2002; Tu et al., 2007; Yun et al., 2019). Importantly, quantifying the precise morphology of tertiary sulci also has translational applications.