## Abstract

How an individual’s unique brain connectivity determines that individual’s cognition, behavior, and risk for pathology is a fundamental question in basic and clinical neuroscience. In seeking answers, many have turned to machine learning, with some noting the particular promise of deep neural networks in modelling complex non-linear functions. However, it is not clear that complex functions actually exist between brain connectivity and behavior, and thus if deep neural networks necessarily outperform simpler linear models, or if their results would be interpretable. Here we show that, across 52 subject measures of cognition and behavior, deep neural networks fit to each brain region’s connectivity outperform linear regression, particularly for the brain’s connector hubs—regions with diverse brain connectivity—whereas the two approaches perform similarly when fit to brain systems. Critically, averaging deep neural network predictions across brain regions results in the most accurate predictions, demonstrating the ability of deep neural networks to easily model the various functions that exists between regional brain connectivity and behavior, carving the brain at its joints. Finally, we shine light into the black box of deep neural networks using multislice network models. We determined that the relationship between connector hubs and behavior is best captured by modular deep neural networks. Our results demonstrate that both simple and complex relationships exist between brain connectivity and behavior, and that deep neural networks can fit both. Moreover, deep neural networks are particularly powerful when they are first fit to the various functions of a system independently and then combined. Finally, deep neural networks are interpretable when their architectures are structurally characterized using multislice network models.

## Main

Human cognitive neuroscience seeks to explain how the function of an individual’s brain determines their behavior. The human brain’s functional connectivity, as commonly measured by the pairwise Pearson’s correlation coefficient between regional time series, has generated tremendous insight into the brain’s network function^{1–19}. A desire to leverage individual variability in these connections to predict cognition, behavior, and symptoms of mental illness has united neuroscientists, clinicians, and machine learning experts^{20–32}. Yet, precisely which prediction algorithm will prove most efficacious is unknown. While deep neural networks display notable predictive power in other domains^{33}, it is not clear whether their application to human brain connectivity is necessary or insightful^{34}. Further, it is not known whether the mathematical functions that define the relationship between brain connectivity and a given outcome are sufficiently complex so as to require deep neural networks instead of simpler linear models^{34,35}. Moreover, even should deep neural networks offer accurate predictions, their interpretability has been rightfully questioned^{20,36–38}.

Here, we use linear regression, support vector machines, and deep neural networks to iteratively predict 52 subject measures, including performance on cognitive tasks and answers to questionnaires, from the connections of each brain region (Figure 1a-c). In the course of this exercise, we are able to determine whether the functional forms of the relations between brain connectivity and observable subject measures are sufficiently complex so as to require deep neural networks. Moreover, instead of assuming that a single function exists between connectivity and behavior, we carve the brain at its joints with deep neural networks, generating a different function between each region’s connectivity and each subject measure. These predictions can then by averaged, and we can test whether that agglomeration increases our prediction accuracy. Finally, deep neural networks are naturally modeled as multislice networks, as each layer of the network is only connected to itself, forming a “slice” of the deep neural network. We can thus optimize a multislice modularity quality index (*Q*) to assess the community structure of the trained deep neural networks (Figure 1a)^{40}. A high *Q* value indicates that the deep neural network is decomposable into communities with dense connectivity within each community and sparse connectivity between communities, indicating mostly independent paths through the deep neural network from the feature input layer to the prediction node. By building and structurally characterizing multislice network models^{8,41} of the trained deep neural networks in the context of which brain region the model is capturing, and how accurate the model is, we are able shine light into the black box of deep neural networks to understand how and why they perform as they do.

We hypothesized that, while the function between connectivity and behavior for many regions and systems can be captured by linear regression, more complex functions that exist at the regional level require deep neural networks. We expect these more complex functions to exist at the brain’s connector hubs^{12}, given their diverse connectivity across the brain’s communities, their cortical expansion from macaques to humans^{41,42}, and their involvement in many different and more complex cognitive processes^{12}. We quantify connector hubs on a continuum by the participation coefficient (*PC*)^{2–4}. More generally, we reasoned that all regions are unlikely to share the same function between brain connectivity and behavior. Thus, we hypothesized that the best predictions across subject measures would be made while using a different deep neural network for each region, generating unique functions linking regional connectivity to behavior, and then averaging those predictions together. Finally, we hypothesized that the architectures of deep neural networks would have to be more modular while modeling connector hubs than while modeling non-connector hubs, in order to disentangle the different non-linear components of the function between connectivity and behavior.

### Deep neural networks outperform linear regression when using regional connectivity

We first sought to measure the performance of linear and deep neural network models to predict subject measures from regional and system level connectivity. For each subject measure in the Human Connectome Project (*n*=52) and for each region in the brain (*n*=400^{11}), we used that region’s functional connections to predict that subject measure with deep neural networks and separately with linear regression (Figure 1a-c, Methods). We employed 5-fold cross-validation such that each prediction is generated by applying the trained model to the connectivity of untrained subjects (Figure 1b, Methods). We considered 100 distinct architecture for the deep neural networks, ranging from 1 to 10 layers and from 10 to 400 neurons (Figure 1c, Methods). We assessed prediction accuracy by calculating the Pearson *r* correlation coefficient between the observed and predicted subject measures. To move beyond regional information, we also considered the 17 “Yeo” cognitive systems that were defined previously^{11}; specifically, we used each system’s connectivity to other systems as the 16 input features to the prediction algorithm. This approach allowed us to determine whether deep neural networks outperform linear regression only when the granularity of the connectivity feature space is high. In sum, for every subject measure (Figure 1d), we quantify how well we can predict that subject measure across 100 deep neural network architectures and separately with linear regression, both using regional connectivity and system connectivity.

Using system connectivity, we observed no significant differences between the prediction accuracy of deep neural networks and linear regression (Figure 2a). However, using regional connectivity, deep neural networks outperformed linear regression by a significant margin (Figure 2b). Next, we investigated which deep neural network predicted subject measures the most accurately, as well as which deep neural network architectures best captured the function between the connectivity of connector hubs and subject measures (Figure 2c,d). When considering either system or regional connectivity, deep neural network architectures with more layers and more neurons more accurately captured the connectivity of connector hubs. Notably, this result was less pronounced for system connectivity, where very small (1 layer, 10 neurons) deep neural network architectures produced accurate predictions for systems with many connector hubs (Figure 2c,d).

### Deep neural networks fit complex connector hubs

Next, we sought to decipher where in the brain deep neural networks are more accurate than linear regression, and vice versa, relative to each model’s mean prediction accuracy. We hypothesized that connector hub regions, given their diverse connectivity across the brain’s communities, would require deep neural networks to discover the function between their connectivity and behavior. We thus measured each region’s mean prediction accuracy across deep neural network architectures and subject measures, and we then *z*-scored those accuracy values. We also *z*-scored each region’s mean prediction accuracy across subject measures for the linear regression models. Our choice to *z*-score was motivated by the goal to obtain a relative measure of where in the brain each model is least versus most accurate.

We found that regions with higher *z*-scored prediction accuracy for linear regression tended to be located in visual, sensory, and motor cortex (Figure 2e), while regions with higher *z*-scored prediction accuracy for deep neural networks tended to be connector hubs located in fronto-parietal cortex (Figure 2f,g). To further determine whether deep neural networks are most accurate at connector hub regions, we calculated the Pearson *r* correlation coefficient between each region’s prediction accuracy and participation coefficient, for each subject measure. We found that the *r* values were significantly larger for deep neural networks than for linear regression (Figure 2h). Moreover, we observed a positive and significant Pearson *r* correlation coefficient between each region’s mean prediction accuracy across subject measures for deep neural networks and each region’s participation coefficient (Figure 2i). This relationship held true even after subtracting the mean prediction accuracy across subject measures for linear regression models (Figure 2j). Taken together, these results demonstrate that deep neural networks, in general and compared to linear regression, generate the most accurate predictions of subject measures from connector hub connectivity.

### Combined deep neural networks best predict behavior

If deep neural networks are carving the brain at its joints, generating the best unique function for each region, then averaging the predictions across regions to generate a single prediction of the subject measure across individuals should be highly accurate. To test this expectation, we generated a combined prediction for each subject measure and for each deep neural network architecture, where the mean prediction across regions for that subject measure is calculated; we calculate the mean, across regions, of the 400 (regions (or 17 systems)) x 607 (subjects) prediction array, generating a 1×607 prediction array. This averaging simultaneously leverages the unique functions at each region and reduces prediction noise from overfitting. In contrast to deep neural networks, we expect a simple linear model to, on average, generate the best predictions from individual systems, because averaging regional connectivity down to system level connectivity removes any complex non-linear functions between connectivity and behavior. To test this expectation, we compared the distributions of prediction accuracies for system and regional predictions across linear and deep neural networks, both from individual systems and regions as well as while averaging the predictions across systems or regions. Recapitulating what we observed above (Figure 2a), we found that the deep neural networks outperform the linear models for regional connectivity (Figure 3a). In contrast, the linear models for system connectivity outperform the deep neural network models for regional connectivity (Figure 3a). Finally, the system models outperform the regional models, both for linear and deep neural networks (Figure 3a). In sum, when making a prediction from a single region or system, the most accurate predictions come from models of system connectivity, with linear models being numerically but not significantly more accurate than deep neural networks.

Notably, whereas the mean accuracy is highest for models of system connectivity, deep neural networks for regional connectivity still generated the highest accuracies we observed, with many *r* values above 0.40 (Figure 3b). In fact, the best prediction accuracies for each subject measure for deep neural networks of regional connectivity were significantly higher than the linear models’ top prediction accuracies for each subject measure (student’s independent *t*=6.60, -log10(*p*)=9, *df*=102). Thus, deep neural networks trained on regional connectivity are able to find the strongest relationships between the brain and behavior, particularly at connector hubs, even though they do not perform optimally, on average, across all regions in comparison to linear models of system connectivity (Figure 3a). As described above, however, we can make a combined average prediction that leverages both this optimality at connector hubs and the ability of deep neural networks to fit a function to every brain region. When making combined average predictions, utilizing information across all regions or systems, deep neural networks applied to regional connectivity perform better than any other model tested here (Figure 3b). For each subject measure, we visualized the distributions of the prediction accuracies, comparing the combined deep neural network prediction accuracies to the most accurate deep neural network architectures and linear models for a given subject measure from a single region. In general, we found that no model could outperform the combined predictions from deep neural networks trained on regional connectivity. Thus, the most accurate model of brain connectivity and subject measures appears to actually be a large collection of deep neural networks, one for each brain region.

Finally, we sought to determine whether deep neural networks are able to predict subject measures better than linear models because they are fitting many unique functions to the relation between brain connectivity and behavior, or if they are simply fitting a single non-linear function. Thus, we executed the same analyses as above—comparing prediction accuracy for system and regional connectivity, as well as combined predictions (Figure 3)—comparing deep neural networks to support vector machines with a sigmoid function (Extended Data Figure 1), a three-degree polynomial function (Extended Data Figure 2), and a radial basis function (Extended Data Figure 3). Deep neural networks were significantly more accurate, both when predicting from a single region or system and when making a combined prediction across regions or systems. In fact, median prediction accuracy for the support vector machines was always below zero, demonstrating that none of these single non-linear functions capture all brain connectivity and behavior well. Thus, the ability of deep neural networks to fit a unique function to each region—be it a linear or non-linear function—may explains their ability to outperform other machine learning models.

In sum, deep neural networks trained on regional connectivity carve the brain at its joints, generating complex, but meaningful, functions that are unique to each region. While deep neural networks best model the connectivity of connector hubs, they can also model other regional connectivity well, and, by combining all of these predictions, we obtain the most accurate predictions of human behavior. Critically, neither of these characteristics exist for linear models.

### Connector hubs are modeled by modular deep neural networks

Next, we sought to understand how the architecture of a deep neural network can support accurate predictions. In particular, we were interested in discovering why deep neural networks are able to model connector hub regions so accurately. We modeled the deep neural network as a multislice network in which each hidden layer’s neurons and connections between them comprise a slice and each neuron is connected to itself across slices. Moreover, in order to isolate phenomena that are driven by training the deep neural networks on brain connectivity and behavior from deep neural networks in general, for each subject measure, we shuffled the values and trained a deep neural network to predict random subject measure values from brain connectivity. Next, we assess the multislice network’s architecture by maximizing a multislice modularity quality index, *Q*, to identify communities of neurons that exists across the slices^{40} (Methods, Figure 1a,c). Here, a community is a collection of neurons that are tightly connected to each other as one traverses the layers of the deep neural network. An intuitive notion of this type of community is a path or stream involving multiple neurons through the layers of the deep neural network, as these neurons remain tightly interconnected across layers. A set of communities would therefore indicate parallel streams of neurons across layers that allow for simultaneous processing of distinct functions.

We found that deeper (more layers) and wider (more neurons) neural networks exhibited higher Q values (Figure 4a).This observation was true of deep neural networks in general, as both deep neural networks trained on the real and shuffled subject measures exhibited this effect. Moreover, the Q value for each architecture was highly similar, regardless of whether the deep neural network was trained on original or shuffled subject measures (Pearson’s r=0.985,-log10(p)>100, df=48; Extended Data Figure 4a). Only when using the real (not shuffled) data, the same deep and wide architectures that exhibited high Q exhibited high prediction accuracy (Figure 4b,d; Extended Data Figure 4b). Interestingly, the most accurate architectures for connector hubs were not necessarily deep, but wide (Figure 4c); again, this effect was not present when the deep neural networks were trained on shuffled subject measures (Extended Data Figure 4c). To investigate this architectural specificity further, we calculated a polynomial regression between each network’s Q and prediction accuracy (Figure 4d). In order to know if modeling a connector hub well drives the deep neural network architecture to be extra modular, we analyzed the residuals from the exponential fit—positive residuals mean that the network is more modular than it has to be, in general across brain regions, in order to be accurate. We found that the residuals from this function were significantly higher for architectures that modeled the connectivity of connector hubs well (Figure 4d,e). Critically, this effect was not present when the deep neural networks were trained on shuffled subject measures (Extended Data Figure 4d,e). Moreover, across subject measures, the modular deep neural network architectures were the most accurate if they also modeled the connectivity of connector hubs well (Figure 4f). Again, this effect was not present when the networks were trained on shuffled subject measures (Extended Data Figure 4f). Thus, it appears that modeling the connectivity of connector hubs drives deep neural networks to be extra-modular.

Finally, connector hubs, because they are widely connected to many communities across the brain, potentially contain information about the entire brain’s connectivity^{2–4,18}. If true, we reasoned that a deep neural network architecture should be able to predict a subject measure quite well from a single region, so long as that region is a connector hub^{2–4,18}. Consistent with our expectation, we found that the deep neural network architectures that were able to most accurately model the connectivity of connector hubs were also able to outperform the combined prediction accuracy measure (Figure 4g,h). These architectures were all wide (Figure 4g). This effect was not present when the deep neural networks were trained on shuffled subject measures (Extended Data Figure 4g,h). Thus, a single region can outperform whole brain predictions, so long as the deep neural network architecture is wide (many neurons) and the region is a connector hub.

## Discussion

We discovered that there are complex relationships between brain connectivity and behavior that are best captured by deep neural networks and we can use network science to interpret the architectural properties of the most accurate deep neural networks. Deep neural networks, particularly ones with many neurons, become modular in order to most accurately model the connectivity of connector hubs. Connector hubs have been demonstrated to integrate information across the brain’s systems, as well as coordinate connectivity between the brain’s systems, allowing for a balance of modular and integrative processing^{2–4,7,16–18,23,42–44}. The connectivity of these regions thus likely integrates behaviorally relevant information from across the entire brain. Using multislice network models to measure the modularity of trained deep neural networks, we find that wide and modular deep neural networks decompose the complex connectivity of connector hubs optimally, likely by creating mostly independent communities or parallel paths through the network to best learn the nonlinear relations between connector hub connectivity and behavior.

Past research has shown that modular biological networks emerge when varying the training goals of the system^{45} and modular neural networks exhibit faster information spreading and more precise memory^{46}. Here, we add that, in the context of brain connectivity and behavior, deep neural networks perform better when their architecture is modular, particularly for the brain’s connector hubs, likely because these regions have diverse connectivity to the brain’ communities. The deep neural network architectures are able to decompose each connector hub’s connectivity to individual communities while also modeling the relevant dependencies between the way in which the connector hub is connected to each community. Potentially because of this modeling ability, predictions from connector hubs are often better than predictions utilizing the entire brain’s connectivity. Despite highly accurate predictions made from connector hubs in certain deep neural network architectures, we found that, in general across deep neural network architectures, the most accurate predictions of behavior come from combining the predictions of all regions.

In sum, instead of requiring a single function to explain the relation between brain connectivity and behavior, deep neural networks are able to carve the brain along multiple unique joints, from simple linear ones to complex nonlinear ones, that exist between the brain and behavior. Critically, our results motivate future studies to determine whether deep neural networks could better decompose the complex functions that exist between brain connectivity and mental illness^{9,10,37,26,47–50}.

## Methods

### Subject Measures

We chose to analyze a large and diverse set of measures in the Human Connectome Project to ensure that our conclusions were valid across distinct assessments of cognition and behavior. We included all measures from the Alertness, Cognition, Emotion, Personality, and Sensory categories. In addition, we used several in-scanner measures of behavioral performance on cognitive tasks. Extended Data File 1 includes the full list of measures and our more natural language names for them.

### Human Brain Connectivity

The Resting-state, Working Memory, Language and Math, Relational, Social Cognition, Emotion, Gambling, and Motor fMRI scans from the S1200 Human Connectome Project release^{51} were analyzed. Because our analyses included many different subject measures, we chose to use all scans in order to sample the brain’s functional connectivity in a diverse range of cognitive states.

Additionally, this choice likely increases the accuracy of our measurement of each functional connection, given the increase in the length of the time series. Brains were normalized to fslr32k via the MSM-AII registration. CompCor, with five principal components from the ventricles and white matter masks, was used to regress out nuisance signals from the time series. In addition, the 12 detrended motion estimates provided by the Human Connectome Project were regressed out from the regional time series. The mean global signal was removed and then time series were band-pass filtered from 0.009 to 0.08 Hz^{52-53}. Finally, frames with greater than 0.2 mm frame-wise displacement or a derivative root mean square (DVARS) above 75 were removed as outliers. Sessions composed of greater than 50 percent outlier frames were not further analyzed. The processing pipeline used here has previously been suggested to be ideal for removing false relations between connectivity and behavior^{46}. We then only analyzed data from subjects that had all scans remaining after scrubbing, leaving 607 subjects.

We used 5-fold cross validation, where the subjects are split into five groups, the model is trained on 80 percent of the subjects (i.e., four groups), and then the model is used to predict the subject measure on the 20 percent of subjects that the model has not seen. In line with previous and similar studies that compare machine learning methods, in order to maximize the number of subjects, twins were included^{2,34}. We note that, critically, we used the same division of subjects into folds for all models, ensuring that any potential performance advantage due to twins in the data was equally present in all prediction accuracy measures. This approach also ensures that differences across predictions are not due to differences in folds.

We parcellated the brain into 400 regions^{11} on the fslr32k cortical surface and calculated the Pearson correlation coefficient between all pairs of regions to generate a 400×400 matrix for each scan; to each element we applied Fisher’s r-to-z transformation (numpy.arctanh). These 18 matrices (four for the resting-state, and two for each task) were then averaged for each subject into a final 400×400 matrix. To calculate the participation coefficient, we took the mean connectivity matrix across all subjects and scans. Across 50 linear steps from thresholds of 90^{th} to 95^{th} percentiles (i.e., graph densities from 0.1 to 0.05), we calculated the weighted participation coefficient, where the canonical 17 systems were used as communities^{11}. The mean participation coefficient across thresholds was then calculated and used for all analyses.

### Machine Learning

We used python’s sklearn library to train deep neural networks, linear models, and support vector machines. For the deep neural networks, we used: sklearn.neural_network.**MLPRegressor** with stock parameters, except for using “ solver=‘lbfgs’ “, as, for small datasets, ‘lbfgs’ can converge faster and perform better (see sklearn documentation). We then trained networks in all combinations of 1-10 layers in steps of 1 and 10, and 10, 15, 25, 50, 75, 100, 150, 200, 300, and 400 neurons. Linear regression was executed with stock parameters as: sklearn.linear_model.**LinearRegression**. For support vector machines, we used: sklearn.svm.**SVR**(*kernel)*, with the kernel set to ‘sigmoid’, ‘poly’(stock setting is 3-degree), or ‘rbf’. For each region and subject measure, we fit a deep neural network, a linear model, and the three versions of the support vector machine. For every subject measure, we used 5-folds cross validation (sklearn.model_selection .KFold(5, shuffle=True) to generate 5 folds, and then we ran each model—deep neural network, linear, and support vector machine—on the same division of subjects into 5 folds, as prediction accuracy can vary if the division of subjects into folds differs. We then measured prediction accuracy as the Pearson *r* correlation coefficient between the predicted and observed subject measure.

### Computing

Deep neural networks are notoriously expensive to train. Here, we levered the CUBIC cluster at the University of Pennsylvania, which consists of 168 compute nodes, providing a total of 4804 CPUs. Each deep neural network model used, at most, 4GB of RAM to run. On this cluster, running all regions (*n*=400), all subject measures (*n*=52) and all architectures (*n*=100), takes approximately one week. Including the multislice modularity analyses, linear models, and system level predictions, the analyses here take approximately two weeks on our cluster, or roughly 1,600,000 hours of compute time.

### Multislice Modularity

A multislice network is composed of individual networks coupled through links that connect each node in one network slice to itself in other slices^{41}. One example of a common multislice network is a network that changes through time, where each point in time is represented as a “slice”. A deep neural network can also naturally be seen as a multislice network, where each hidden layer is a slice, and weights link nodes to one another, forming a neuron-by-neuron connectivity matrix. Moreover, each neuron is connected to itself in the previous and next layer. In order to model the strongest edge weights in the deep neural network, the matrix is thresholded to only retain weights above the 90^{th} percentile, which is the median of our typical set of thresholds^{2–4}, while ensuring that every node has at least one edge by calculating the maximum spanning tree^{2}. Edge weights are not binarized. In contrast to when we predict subject measures with 5-fold cross validation, here the deep neural network is trained on the full set of subjects in order to generate a single deep neural network that models the function between brain connectivity and the given subject measure for all subjects. This is also necessary, as we sought to analyze a single deep neural network for each region and subject measure. For multislice modularity^{40}, where we detect communities across slices, each neuron is linked to itself across slices by the Ω parameter, which is set to 0.5. We chose this value because the mean weight in the multislice network is roughly 1, so a Ω of 0.5 maintains the node’s connectivity to itself throughout layers while allowing the node to change communities. Finally, γ is a resolution parameter that determines the resolution at which communities are detected, which is set to 2. We chose this value to ensure that no neural network produced less than three communities, nor was the network partitioned such that each node is its own community (for examples, see Figure 1c).

Although each parameter value was reasonably chosen to result in interpretable partitions of deep neural network nodes into communities, ideally, all three parameter choices, as well as a range of graph thresholds, are tested across a range of similarly reasonable choices. However, these parameters had to be chosen *a priori*, because, even with our current computing power, this is simply not feasible.

## Code and Data Availability

Subject matrices and full analysis code can be found at: github.com/mb3152/deep_prediction.

## Extended Data Figures

## Citations

### Citation Diversity Statement

Recent work in neuroscience and other fields has identified a bias in citation practices such that papers from women and other minorities are under-cited relative to the number of such papers in the field^{52–54}. Here we sought to proactively consider choosing references that reflect the diversity of the field in thought, form of contribution, gender, race, geographic location, and other factors. We used open-source code^{55} that uses an automatic classification of gender (gender-api.com) based on the first names of the first and last authors, with possible combinations including man-man, man-woman, woman-man, woman-woman. Excluding self-citations to the first and senior authors of our current paper, the references contain 57% (n=25) man-man, 16% (n=7) man-woman, 16% (n=7) woman-man, and 11% (n=5) woman-woman. Expected proportions in the top 5 neuroscience journals, as reported in Dworkin et al^{54} are, respectively: 58.4%, 9.4%, 25.5%, and 6.7%. We look forward to future work that could help us to better understand how to support equitable practices in science.