Functional brain network reconfiguration during learning in a dynamic environment
=================================================================================

* Chang-Hao Kao
* Ankit N. Khambhati
* Danielle S. Bassett
* Matthew R. Nassar
* Joseph T. McGuire
* Joshua I. Gold
* Joseph W. Kable

## Abstract

When learning about dynamic and uncertain environments, people should update their beliefs most strongly when new evidence is most informative, such as when the environment undergoes a surprising change or existing beliefs are highly uncertain. Here we show that modulations of surprise and uncertainty are encoded in a particular, temporally dynamic pattern of whole-brain functional connectivity, and this encoding is enhanced in individuals that adapt their learning dynamics more appropriately in response to these factors. The key feature of this whole-brain pattern of functional connectivity is stronger connectivity, or functional integration, between the fronto-parietal and other functional systems. Our results provide new insights regarding the association between dynamic adjustments in learning and dynamic, large-scale changes in functional connectivity across the brain.

## Introduction

Human decisions are guided by beliefs about current features of the environment. These beliefs often must be inferred from indirect and uncertain evidence. For example, deciding to go to a restaurant typically relies on a belief about its current quality, which can be inferred from past experiences at that restaurant. This inference process is particularly challenging in dynamic environments whose features can change unexpectedly. For example, the current quality of the restaurant might not reflect its past quality if a new chef was just hired. In these environments, people tend to follow normative principles and update their beliefs dynamically and adaptively, such that new evidence is used to update existing beliefs more strongly in some conditions (e.g., when existing beliefs are weak or irrelevant, and/or the new evidence is strong or surprising) than others (e.g., when existing beliefs are strong and relevant, and/or the new evidence is weak or expected)1, 2, 3. A number of studies have begun to identify potential neural substrates of this adaptive belief-updating process, including univariate and multivariate activity patterns that encode key variables like uncertainty and surprise in a number of brain regions including parts of the dorsomedial frontal cortex, anterior insula, lateral prefrontal cortex, and lateral parietal cortex2, 4, 5, 6, 7. The goal of the present study was to gain deeper insights into how these representations might interact dynamically and adaptively to support dynamic and adaptive belief updating.

Here we focused on how changes in belief updating relate to changes in functional connectivity between brain regions with task-relevant activity modulations. Functional connectivity reflects statistical dependencies between regional activity time series8 and can be used to build functional-connectivity networks that can provide new perspectives on brain function9, 10, 11. Many recent studies of learning have focused on brain network reconfigurations that occur between naïve and well-learned phases in various domains such as motor, perceptual, category, spatial, or value learning12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22. In these cases, functional connectivity associated with the fronto-parietal system decreased gradually as learning progressed and this change in connectivity was associated with individual learning or performance13, 19, 22. In dynamic environments, however, people progressively learn the current state and then re-initialize learning once the state changes. Thus, we expected frequent reconfigurations in functional connectivity, as learning shifts between slower and faster updating in response to changes in uncertainty and surprise. Additionally, although brain regions that encode uncertainty and surprise participate in multiple networks, including the fronto-parietal task control system, the dorsal attention system, the salience system, and the memory system2, 4, 5, 6, 7, based on previous network analyses of learning in stable environments we hypothesized that the fronto-parietal system would serve a particularly important role in network reconfiguration during learning in dynamic environments.

In the current study, we aimed to identify such frequent reconfigurations in functional connectivity during adaptive belief updating. A key to our approach was the use of an unsupervised machine-learning technique known as non-negative matrix factorization (NMF)23. NMF decomposes the whole-brain network into subgraphs, which describe patterns of functional connectivity across the entire brain, and the time-dependent magnitude with which these subgraphs are expressed. Briefly, a subgraph is a weighted pattern of functional interactions that that statistically recurs as the brain network evolves over time. We chose NMF because it provides two key advantages over other approaches to matrix factorization, such as principal components analysis (PCA) or independent components analysis (ICA)24, 25. First, NMF yields aparts-based representation of the network, in which the individual components are strictly additive – a constraint that is not present in PCA and ICA. This important feature enables interpretation of the resulting subgraph and time-dependent expression coefficients on the basis of their positive distance from zero. Second, NMF does not enforce an orthogonality or independence constraint and, therefore, allows subgraphs to overlap in their structure. This property may more effectively model distinct subgraphs that may be jointly related via weak connections and better account for the flexibility of neural systems, such that one connection between regions can be involved in multiple systems or cognitive functions. Recently, NMF has been used to identify network dynamics during rest and task states25, 26 and to determine how these dynamics vary across development24. Here we extend the use of this technique to examine changes in network dynamics linked to task variables and individual differences.

Our primary hypothesis was that uncertainty and surprise, task variables that drive the adjustment of learning, are related to the temporal expression of specific patterns of functional connectivity (i.e., specific subgraphs). Given previous univariate and multivariate results on learning in dynamic environments, and previous network results on learning in stable environments, we expected these patterns of functional connectivity to prominently involve the fronto-parietal network. We also expected that the dynamic modulation of these patterns of functional connectivity (i.e., subgraph expression) are associated with individual differences in learning.

## Results

### Belief updating is influenced by normative factors related to uncertainty and surprise

Participants performed a predictive-inference task during fMRI (Fig. 1a). For this task, participants positioned a bucket to catch a bag that dropped from an occluded helicopter. The location of the bag was sampled from a Gaussian distribution with a mean value that corresponded to the location of the helicopter and a standard deviation (noise level) that was set to either high or low in each 120-trial run. The location of the helicopter usually remained stable but occasionally changed suddenly and unpredictably (with an average probability of change of 0.1 across trials); the new location was sampled from a uniform distribution. Additionally, whether the bag (if caught) was rewarded or neutral was assigned randomly on each trial and indicated by color. This task challenged participants to form and update a belief about a latent variable (the location of the helicopter) based on noisy evidence (the location of dropped bags).

![Fig. 1](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2020/01/17/800284/F1.medium.gif)

[Fig. 1](http://biorxiv.org/content/early/2020/01/17/800284/F1)

Fig. 1 Overview of the task and theoretical model of belief updating (McGuire et al., 2014).
**a** Sequence of the task. At the start of each trial, participants predict where a bag will drop from an occluded helicopter by positioning a bucket on the screen. After participants submit their prediction, the bag drops and any rewarded coins that fall in the bucket are added to the participant’s score. The location of the last prediction and the last bag drop are noted on the next trial. **b** An example sequence of trials. Each data point represents the location of a bag on each trial (yellow for rewarded coins, gray for neutral coins). The dashed line represents the true generative mean. The mean changes occasionally. The cyan line represents the prediction from a normative model of belief updating. The inset equation shows how the model updates beliefs (*B*t = belief, *X*t = observed outcome, αt = learning rate on trial *t*). The vertical dashed line represents the boundary of the noise conditions: high-noise (left) and low-noise condition (right). Noise refers to the variance of the generative distribution. **c** Two learning components from the normative model. Change-point probability (CPP) reflects the likelihood that a change-point happens, which is increased when there is an unexpectedly large prediction error. Relative uncertainty (RU) reflects the uncertainty about the generative mean relative to the environmental noise, which is increased after high CPP trials and decays slowly as more precise estimates of the generative mean are possible. The inset formula shows how CPP and RU contribute to single trial estimates of learning rates.

In our previous reports2, 7, we described a theoretical model approximating the normative solution for this task. This theoretical model takes the form of a delta-rule and approximates the Bayesian ideal observer. Beliefs (*B*t+1) are updated based on past beliefs (*B*t) and prediction errors (*X*t − *B*t; the difference between the current outcome location and the predicted location), with the extent of updating controlled by a learning rate (αt; Fig. 1b). Learning rates vary from trial to trial and are determined by two factors: (i) change-point probability (CPP), which is the probability that a change-point has happened and represents a form of belief surprise; and (ii) relative uncertainty (RU), which is the reducible uncertainty regarding the current state relative to the irreducible uncertainty that results from environmental noise and represents a form of belief uncertainty (Fig. 1c). Learning rates are higher when either CPP or RU is higher: αt = *CPP* + *RU* (1 − *CPP*).

We previously reported how participants’ predictions were influenced by both normative and non-normative factors and how these factors that influence learning are encoded in univariate and multivariate activity2, 7. Participants updated their beliefs more when the value of CPP or RU was higher, consistent with the normative model. Participants also updated their beliefs more when the outcome was rewarded, however, which is not a feature of the normative model. CPP, RU and reward, as well as residual updating (belief updating not captured by CPP, RU or reward), were all encoded in univariate and multivariate brain activity in distinct regions2, 7. In the current study, we built on these previous findings and investigated how these factors, as well as individual differences in how these factors influence belief updating, are related to the dynamics of whole-brain functional connectivity.

### NMF identified ten subgraphs, or patterns of whole-brain functional connectivity, that varied over time

We used NMF to decompose whole-brain functional connectivity over time into specific patterns of functional connectivity, called subgraphs, and quantified the expression of these patterns over time. To perform NMF, we first defined regions of interest (ROIs) based on a previously defined parcellation27 (Fig. 2a) and extracted blood-oxygenation-level-dependent (BOLD) time series for each ROI (Fig. 2b). For every pair of ROIs, we calculated the Pearson correlation coefficient between the BOLD time series in 10-TR (25 s) time windows, offset by 2 TRs for each time step (and thus 80% overlap between consecutive time windows). This procedure thus yielded a matrix whose entries represented time-dependent changes in the strengths of these pairwise correlations in the brain during the task. We unfolded each time window from this correlation matrix (Fig. 2c) into a one-column vector, and then concatenated these vectors from all time windows and all participants (Fig. 2d). As required for NMF, we transformed the resulting matrix to have strictly non-negative values: we duplicated the full matrix, set all negative values to zero in the first copy, and set all positive values to zero in the second copy before multiplying all remaining values by negative one. Thus, we divided the final full data matrix into two halves, with one half containing the positive correlation coefficients (zero if the coefficient was negative) and one half containing the absolute values of the negative correlation coefficients (zero if the coefficient was positive)26. This procedure ensured that our approach did not give undue preference to either positive or negative functional connectivity, and that subgraphs were identified based on both positive and negative functional connectivity.

![Fig. 2](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2020/01/17/800284/F2.medium.gif)

[Fig. 2](http://biorxiv.org/content/early/2020/01/17/800284/F2)

Fig. 2 Schematic overview of the method.
**a** Regions of interest (ROIs). Functional MRI BOLD signals were extracted from spherical ROIs based on the coordinate system of Power et al. (2011). There were 264 ROIs in this coordinate system. We only kept ROIs that had usable data from all subjects. Thus, we used 247 ROIs for the following analyses. Each ROI can be assigned to one of 13 putative functional systems. **b** An example of Pearson correlation coefficients calculated between regional BOLD time series over the course of the experiment. Each BOLD time series was divided into 10-TR (25 seconds) time windows, and consecutive time windows were placed every 2 TRs leading to 80% overlap between consecutive time windows. Pairwise Pearson correlation coefficients were calculated between ROI time series in each time window. **c** An example of edge strength over time. In each time window, there were 247*(247-1)/2 edges. **d** Nonnegative matrix factorization (NMF). In each time window, the matrix of edge strengths was unfolded into one column. Then, edges from all time windows in all participants were concatenated into a single matrix. Each row in the full data matrix contained an edge (pairwise correlation coefficients between BOLD time series from two ROIs) and each column contained a time window (across all scans and participants). Correlation values in this matrix were strictly non-negative; the full data matrix was divided into two halves, with one half containing the positive pairwise correlation coefficients (zero if the correlation coefficient was negative) and one half containing the absolute values of negative pairwise correlation coefficients (zero if the correlation coefficient was positive). Thus, subgraphs were identified based on both the similarity of positive functional connectivity and the similarity of negative functional connectivity together. Then, NMF was applied to decompose the concatenated matrix into a matrix ***W***, which encoded the strengths of edges for each subgraph, and a matrix ***H***, which encoded the time-dependent expression of each subgraph. For example, the strength of edges of the fourth subgraph (the fourth column in the matrix ***W***) can be folded into a squared matrix, reflecting the edge strength between every pair of ROIs.

We applied NMF to this matrix (**A**) to identify functional subgraphs and their expression over time. Specifically, we decomposed the full data matrix into a subgraph matrix ***W*** and an expression matrix ***H*** (Fig. 2d). The columns of ***W*** represent different subgraphs and the rows represent different edges (i.e., pairs of regions), with the value in each cell representing the strength of that edge (i.e., the functional connectivity strength for that pair of regions) for that subgraph. The rows of ***H*** represent different subgraphs, and the columns represent time windows, with the value in each cell representing the degree of expression of that subgraph in that time window. We implemented NMF by minimizing the residual error ![Graphic][1]</img> via three parameters: (i) the number of subgraphs (*k*), (ii) the subgraph regularization (*α*), and (iii) the expression sparsity (*β*) (Supplementary Fig. 1). The final result of NMF is a set of subgraphs (the number of which was determined by the parameter *k*), which reflected patterns of functional connectivity strengths across every pair of regions in the brain, as well as the expression of these subgraphs over time.

Using NMF, we identified ten subgraphs in our data. The full description of each subgraph specifies the edge strength between every pair of ROIs, corresponding to a 247×247 matrix. We calculated a simpler summary description that specifies the edge strength between every pair of functional systems in the previously defined parcellation, corresponding to a 13×13 matrix27. Edges between ROIs were categorized according to the functional system of each ROI. To estimate the diagonal entries in the system-by-system matrix, we averaged the weights of all edges connecting two ROIs within a given system (Fig. 3a). To estimate the off-diagonal entries of the system-by-system matrix, we averaged the weights of all edges linking an ROI in one system with an ROI in another system. In line with common parlance, we refer to the edges within the same system as within-system edges, whereas we refer to the edges between two different systems as between-system edges. For presentation, we ordered and numbered the ten subgraphs according to the strength of within-system edges relative to that of between-system edges (Fig. 3b, Supplementary Fig. 2a-c). Finally, we thresholded the system-by-system matrix to show only edges that passed a permutation test (*p*<0.05 after the Bonferroni correction for multiple comparisons; see Methods).

![Fig. 3](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2020/01/17/800284/F3.medium.gif)

[Fig. 3](http://biorxiv.org/content/early/2020/01/17/800284/F3)

Fig. 3 Patterns of connectivity in subgraphs.
**a** Converting edges between nodes into edges between systems. First, the edges of each subgraph can be folded into a square matrix, representing the edges between every pair of nodes (ROIs). Then, based on the 13 putative functional systems reported by Power et al. (2011), we categorized each edge according to the system(s) to which the two nodes (ROIs) belonged. We calculated the mean strength of edges linking a node in one system to a node in another system, and refer to that value as the between-system edge. Similarly, we calculated the mean strength of edges linking two nodes that both belong to the same system and refer to that value as the within-system edge. Edges between nodes and edges between systems were normalized into the scale between 0 and 1. **b** Edges between systems in the ten subgraphs identified by NMF. For each subgraph, the top matrix shows the significant edges in that subgraph within or between systems. We show only significant edges (*p*<0.05 after the Bonferroni correction for multiple comparisons). All nodes from systems involved in significant edges are shown on the brain below. Subgraphs varied in the degree to which they represent interactions within the same system (e.g., subgraph 1) versus interactions between different systems (e.g., subgraph 10).

Figure 3b shows the ten subgraphs identified in our data. Two examples have relatively straightforward interpretations. Subgraph 1 has high edge strengths along the diagonal. Thus, this subgraph describes functional connectivity that lies predominantly within functional systems. In contrast, subgraph 5 has high edge strengths along a single row and column, corresponding to the visual system. Thus, this subgraph describes functional connectivity that predominantly links the visual system with all other systems. Many of the other subgraphs have more complex patterns than these two examples.

The full data matrix on which we performed NMF was divided into two halves, with the first half corresponding to positive functional connectivity and the second half corresponding to negative functional connectivity. The expression matrix ***H*** was therefore also divided into two halves, with the first half corresponding to positive expression over time and the second half corresponding to negative expression over time. Positive and negative expression coefficients were highly negatively correlated with each other across time for all the subgraphs (all *r*<−0.61, all *p*<0.001). Given this result, for the analyses of subgraph expression below, we constructed a measure of relative subgraph expression by subtracting the negative expression from the positive expression at each time point26. Across subgraphs, the average relative expression across time was strongly correlated with the relative strength of within-versus between-system edges (Supplementary Fig. 2d-f). That is, higher within-system strength was associated with greater relative expression of the subgraph. This result implies that the subgraphs are also numbered roughly in order of their average relative expression.

### Normative factors that drive belief updating modulated the temporal expression of a learning-related subgraph

We next investigated how CPP, RU, reward, and residual updating influenced the temporal expression of each subgraph. We identified a particularly strong relationship between the normative factors (CPP, RU and the residuals that reflected the participants’ subjective estimates of those variables) and subgraph 4, whose strongest edges were in the fronto-parietal task-control system, followed by the memory retrieval, salience and dorsal-attention systems (Fig. 4d). Specifically, we used multiple regression to estimate the trial-by-trial relationship between these four factors and the relative expression strength of each subgraph. Regression coefficients were fitted separately for each participant and each subgraph. Then, the significance of these coefficients was tested at the group level for each subgraph (Supplementary Fig. 3). Among the ten subgraphs, these four factors explained the most variance in the time-dependent relative expression of subgraph 4 (Supplementary Fig. 2g), in each case showing positive modulations (CPP: mean±SEM=0.202±0.053, *t31*=3.78, *p*<0.001; RU: 0.392±0.077, *t31*=5.11, *p*<0.001; residual updating: 0.177±0.079, *t31*=2.23, *p*=0.033; Fig 4a). We also evaluated the influence of head motion by including motion, as indexed as the relative root-mean-square of the six motion parameters, in the regression model. Motion was not significant (*p*=0.29) and the effects of CPP, RU and residual updating remained significant and of similar effect size.

![Fig. 4](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2020/01/17/800284/F4.medium.gif)

[Fig. 4](http://biorxiv.org/content/early/2020/01/17/800284/F4)

Fig. 4 Temporal expression of subgraph 4 was related to task factors and individual differences.
**a** Modulation of temporal expression of subgraph 4 by task factors. A regression model that included CPP, RU, reward and residual updating as predictors of temporal relative expression (calculated by subtracting negative expression from positive expression) of subgraph 4 was fitted for each participant, and coefficients were tested on the group level. The results showed positive effects of CPP, RU and residual updating. Error bars represent one SEM. (**p*δ0.05, \***|*p*<0.001) **b** The relationship between individual normative learning and the dynamic modulation of subgraph 4 expression by normative factors. This dynamic modulation was indexed as the sum of the coefficients of CPP and RU in **a**, and represents the extent to which trial-by-trial expression was influenced by the two normative learning factors. There was a significant positive correlation across participants. Each point represents one participant. The red line represents the regression line and the shaded area represents the 95% confidence interval. **c** The relationship between individual normative learning and average relative expression of subgraph 4. There was a significant positive correlation across participants. Each point represents one participant. The red line represents the regression line and the shaded area represents the 95% confidence interval. Source data of **a**-**c** are provided as a Source Data file. **d** Summary of the pattern of connectivity in subgraph 4. We summarized the pattern of connectivity as within-system strength (which is the value in the diagonal) and between-system strength (which is the average of values in the off-diagonal) for each system. The fronto-parietal system as well as three other systems (memory retrieval, salience and dorsal attention) showed the strongest contributions to this subgraph in terms of both within-system and between-system strength. The 95% confidence interval of each system was estimated by boostrapping 10,000 times on the edges of that system. **e** Nodes for the top four systems with strong within-system and between-system strength. We showed the nodes of fronto-parietal system, memory retrieval system, salience system and dorsal attention system on the brain.

Although CPP or RU also modulated the relative expression of some other subgraphs (e.g., subgraph 1, 3 and 7; Supplementary Fig. 3), below we focus on subgraph 4 for several reasons. First, as stated above, the four factors we investigated (CPP, RU, reward and residual updating) explained more variance in the time-dependent relative expression of subgraph 4 than that of any other subgraph. Second, the effects of CPP and RU on subgraph 4 were strong enough to survive correction for multiple comparisons across ten subgraphs, whereas the effects of CPP and RU on other subgraphs were not. Third, as discussed below (see the section of “Results are robust to the time window used to identify functional connectivity“), the effects of CPP and RU on subgraph 4 were also robust across analyses that used differently sized time windows, whereas the effects of CPP and RU on other subgraphs were not.

### Expression of the learning-related subgraph varied across subjects with individual differences in normative belief updating

The expression of subgraph 4 was not only modulated by task factors that drive normative learning, but also varied across subjects in a manner that reflected individual differences in normative learning. As an index of normative learning, we estimated the influence of CPP and RU on trial-by-trial belief updates using multiple regression and took the sum of the regression coefficients of CPP (*β*3 in Equation 6) and RU (*β*5 in Equation 6) for each participant2. This sum reflected how much each individual updated their beliefs in response to normative factors. We examined the relationship between individual differences in this normative belief-updating metric and two aspects of subgraph expression.

First, we examined the relationship between normative belief updating and the dynamic modulation of subgraph expression by normative factors (Supplementary Fig. 4). As an index of the dynamic modulation of subgraph expression by normative factors, we used the sum of the regression coefficients of CPP and RU on relative subgraph expression from the analyses above (Supplementary Fig. 3). We found a positive correlation between the dynamic modulation of subgraph 4 expression by normative factors and normative belief updating across participants (*r*=0.448, *p*=0.004; Fig. 4b). Second, we examined the relationship between normative belief updating and the average relative subgraph expression (Supplementary Fig. 5). We also found a positive correlation between the average relative expression of subgraph 4 and normative belief updating across participants (*r*=0.332, *p*=0.029; Fig. 4c). These effects were still significant when we controlled for the influence of motion on dynamic modulation or average relative expression, whereas the effects of motion itself were not significant (all *p*>0.31).

These two results show that participants with the highest average relative expression of subgraph 4, and for whom the normative factors account for the most variance in the relative expression of subgraph 4 across time, tended to update their beliefs in a manner more consistent with the normative model than the other subjects.

### Contribution of specific edges within the learning-related subgraph to its relationship with task factors and individual differences

Subgraph 4 describes both within- and between-system functional connectivity for multiple different functional systems (Figs. 3b and 4d-e, Supplementary Fig. 2a-c). We next examined the contribution of specific edges (i.e., functional connectivity between specific pairs of brain regions) within subgraph 4 to the task and individual difference effects we observed for that subgraph.

The task-related modulations of subgraph 4 involved primarily between-system, not within-system, functional connectivity. Specifically, we re-estimated the effects of CPP, RU, reward, and residual updating on the relative expression of subgraph 4 using only within-system edges (i.e., only the diagonal cells of the system-by-system matrix in Fig. 3b; “Within”) or only between-system edges (i.e., only the off-diagonal cells of the system-by-system matrix in Fig. 3b; “Between”). We compared these effects to our previous estimates using all edges (Fig. 5a; “All”). Removing the between-system edges (Within versus All) reduced the size of the estimated effects of CPP (mean±SEM=−0.155±0.042, *t31*=−3.73, *p<*0.001), RU (−0.300±0.062, *t31*=−4.82, *p*<0.001), and residual updating (−0.140±0.053, *t31*=−2.63, *p*=0.013). In contrast, removing the within-system edges (Between versus All) led to no reliable changes in these effects (all *p*>0.21). Further, in a direct comparison of the reduced subgraphs with only within-or between-system edges, the effects estimated with between-system edges only were stronger for CPP (0.151±0.042, *t31*=3.63, *p*<0.001), RU (0.290±0.063, *t31*=4.63, *p*<0.001), and residual updating (0.139±0.048, *t31*=2.91, *p*=0.007).

![Fig. 5](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2020/01/17/800284/F5.medium.gif)

[Fig. 5](http://biorxiv.org/content/early/2020/01/17/800284/F5)

Fig. 5 The contribution of between-system and within-system edges to effects of task factors and individual differences on subgraph 4 expression.
**a** The contribution of between-system and within-system edges to the effect of task factors on temporal relative expression of subgraph 4. To determine the relative contribution of between- and within-system edges on time-dependent subgraph 4 expression, we performed three comparisons on the effects estimated by different types of edges: within-system edges only (Within), between-system edges only (Between) and all edges (All). First, removing between-system edges (Within versus All) decreased the effect of CPP, RU and residual updating. Second, in contrast, after removing within-system edges (Between vs All), there was no significant change in these coefficients. Third, we directly compared the effects contributed from between-system edges only and from within-system edges only (Between versus Within). For between-system edges, there were stronger positive effects for CPP, RU and residual updating. Error bars represent one SEM. (**p*<0.05, ***p*<0.01, \***|*p*<0.001) **b** The contribution of between-system and within-system edges to the relationship between normative learning and dynamic modulation and average expression of subgraph 4. We performed the same three comparisons to determine the relative contribution of between- and within-system edges for each relationship with individual differences. For the effect of dynamic modulation, removing within-system edges (Between versus All) decreased the correlation coefficient. This correlation coefficient was also larger for within-system edges only than between-system edges only, but this effect was not statistically significant. For the effect of average expression, removing between-system edges (Within versus All) decreased the correlation coefficient, and the correlation coefficient was larger for between-system edges only than within-system edges only, though neither of these effects were statistically significant. Source data are provided as a Source Data file. Error bars represent one SEM. (***p*<0.01)

The contributions of within- and between-system functional connectivity to the individual difference effects of subgraph 4 were less clear. For the relationship between individual differences in normative learning and average relative expression, the pattern across comparisons was similar to that observed for task effects (Fig. 5b), which would indicate a greater contribution of between-system edges, but none of the comparisons were statistically significant. In contrast, for the relationship between individual differences in normative learning and the dynamic modulation of subgraph 4, within-system edges appeared to be more important, as removing the within-system edges (Between versus All) reduced this correlation (difference = 0.048, *p*=0.006; Fig. 5b).

We also identified contributions of specific functional systems and of specific system-by-system edges to the task and individual difference effects on subgraph 4. Specifically, we conducted similar analyses to the ones described above after removing all edges of one functional system (i.e., one row/column from the system-by-system matrix in Figure 3b; Supplementary Fig. 6) or removing one system-by-system edge (i.e., one cell from the system-by-system matrix in Figure 3b; Supplementary Fig. 7). Detailed results from these analyses are provided in the supplementary materials, but generally the strongest functional systems or individual edges in subgraph 4 (Fig. 4d) contributed the most to the task and individual difference effects described above.

### Results are robust to the time window used to identify functional connectivity

For the analyses above, we calculated functional connectivity within time windows of 25 seconds (10 TRs), with an overlap between adjacent windows of 20 seconds (8 TRs). To determine the sensitivity of our results to the size of this time window, we repeated the entire procedure using shorter (8-TR/20 s window with 6-TR/15 s overlap; Supplementary Fig. 8-11) or longer (12-TR/30 s window with 10-TR/25 s overlap; Supplementary Fig. 12-15) time windows. That is, we shorten or lengthen the time window by the interval of one trial (~5 s). With both shorter and longer time windows, we identified ten subgraphs. As a measure of the similarity between these subgraphs and the subgraphs identified in the main analysis, we calculated the Pearson correlation coefficient between the edge strengths of the subgraphs, for both edges between ROIs and edges between systems. There was a high degree of similarity between the ten subgraphs identified in the main analysis and those identified using either shorter (edges between nodes: all *r*>0.81; edges between systems: all *r*>0.80) or longer (edges between nodes: all *r*>0.98; edges between systems: all *r*>0.98) time windows. With longer time windows, the relative expression of subgraph 4 still showed the same relationship to task factors (CPP and RU) and to individual differences in normative learning; with shorter time windows, these effects were also present but weaker.

### Relationship between the learning-related subgraph and univariate activity

In our previous report, we described how CPP, RU, reward, and residual updating influenced univariate brain activity. In a final set of analyses, we examined the relationship between these previously reported univariate effects and the changes in dynamic functional connectivity we identified above.

The brain regions that were most strongly represented in subgraph 4 overlapped spatially with the brain regions whose activity was modulated reliably by CPP and RU in our previous report. As a measure of a region’s involvement in subgraph 4, for each ROI, we calculated the mean strength of every edge between that ROI and all other ROIs in subgraph 4, and normalized these mean values between 0 and 1. We then related this metric to activation from our previous study2, as measured by the *z*-statistic of the modulation effect of CPP or RU. This z-statistic indicated the effect size of change of univariate activity in response to CPP or RU across participants. Across all ROIs, there was a positive correlation between edge strength in subgraph 4 and activation for CPP (*r*=0.403, *p*<0.0001; Fig. 6a) and activation for RU (*r*=0.704, *p*<0.0001; Fig. 6b). Figure 6c shows the map of normalized mean edge strengths for subgraph 4 alongside the thresholded activation maps for CPP and RU. Regions with stronger edge strength in subgraph 4, such as the insula, dorsomedial frontal cortex, dorsolateral prefrontal cortex, posterior parietal cortex, and occipital cortex, also tended to show stronger increases in activation with increases in CPP and RU.

![Fig. 6](http://biorxiv.org/https://www.biorxiv.org/content/biorxiv/early/2020/01/17/800284/F6.medium.gif)

[Fig. 6](http://biorxiv.org/content/early/2020/01/17/800284/F6)

Fig. 6 Relationship between edge strength of subgraph 4 and univariate task activations.
**a** Relationship between the activation for CPP and the edge strength of subgraph 4. We calculated the Pearson correlation coefficient between the *z*-statistic for CPP from McGuire et al. (2014) and the edge strength across nodes in subgraph 4. Each data point represents an ROI. The edge strength for each ROI was calculated as the column sum of that ROI’s edges to other ROIs, reflecting the summed interactions between that ROI and all others. The edges were normalized into the scale between 0 and 1. A significantly positive correlation was observed. The red line represents the regression line and the shaded area represents the 95% confidence interval. **b** Relationship between the activation for RU and the edge strength of subgraph 4. We observed a significant positive correlation between the *z*-statistic for RU from McGuire et al. (2014) and the edge strength across nodes in subgraph 4. The red line represents the regression line and the shaded area represents the 95% confidence interval. Source data of **a**-**b** are provided as a Source Data file. **c** Whole-brain thresholded activation maps for CPP and RU from McGuire et. al (2014) and whole-brain maps for edge strength of subgraph 4 in the current study.

In addition to these strong associations between univariate brain activation and edge strength, effects beyond those captured by univariate task activity also contributed to our dynamic functional connectivity results. To demonstrate this, we first fit a general linear model to quantify effects of CPP, RU, reward, and residual updating on univariate brain activity. For each ROI, we then constructed a time series of predicted BOLD activity from this GLM. This predicted BOLD time series captured all fluctuations in activity in that ROI that could be accounted for by the linear effects of CPP, RU, reward, and residual updating. Next, we estimated the matrix of functional connectivity from this predicted task-modulated BOLD time series. This matrix captures only the aspects of functional connectivity that can be fully explained by task-modulated co-activation across brain regions. We then performed NMF on this matrix and repeated all of our main analyses, including task effects and individual difference effects (Supplementary Fig. 16-19). This analysis again identified a subgraph 4 whose strongest edges were in the fronto-parietal system, but it did not recapitulate all of the relationships between subgraph 4 expression and task factors and individual differences seen in our main analyses. Namely, with regard to task effects, only RU significantly modulated the temporal expression of the subgraph, and with regard to individual-difference effects, only the relationship between individual differences in normative learning and average relative expression was significant. These results implied that the dynamic functional connectivity patterns identified in our main analyses reflect a mixture of coordinated activity across regions (which can be captured by univariate analyses) and other statistical dependencies across regions that require network-based analyses.

## Discussion

We identified a pattern of dynamic functional brain connectivity in human subjects performing a predictive-inference task. This pattern was expressed most strongly during times that demanded faster belief updating and was enhanced in individuals who most effectively used adaptive belief updating to perform the task. To identify this pattern, we used NMF, an unsupervised machine-learning technique that decomposes the full matrix of time-dependent functional connectivity into subgraphs (patterns of functional connectivity), and the time-dependent magnitude of these subgraphs. Among the subgraphs we identified in our data, the expression of one subgraph in particular was modulated reliably by three trial-by-trial factors that influenced the degree of behavioral belief updating: CPP (surprise), RU (uncertainty), and residual updating (updating unaccounted for by surprise or uncertainty). Notably, CPP and RU are factors that normatively promote greater belief updating, scaling the degree to which past observations are discounted relative to the most recent evidence. Residual updating likely captures, at least in part, deviations between the objective values of CPP and RU in the normative model and the individual’s subjective estimates of those factors. Thus, the expression of this subgraph reflects not only normative factors that should influence belief updating but also likely fluctuations in subjective estimates of those factors. In addition to being modulated by these trial-by-trial task factors, expression of this subgraph also varied across individuals in a manner associated with individual differences in belief updating. Participants who tended to update their beliefs in a more normative manner – that is, with a stronger influence of surprise (CPP) and uncertainty (RU) – showed stronger dynamic modulation of the expression of this subgraph by normative factors and showed stronger average expression of this subgraph.

### Relationship to previous results: neural activation during learning in dynamic environments

The brain-wide pattern of functional connectivity that we identified as connected to belief updating included interactions between multiple functional systems, most prominently the fronto-parietal task control, memory-retrieval, salience, and dorsal-attention systems (Fig. 3b & Fig. 4d). These systems include multiple regions in the anterior insula, dorsolateral and dorsomedial frontal cortex, and lateral and medial parietal cortex (Fig. 4e & Fig. 6c). These regions showed a large degree of overlap with areas that we have previously shown to have increased univariate activation in response to both surprise and uncertainty (in this same dataset; Figure 6)2. The current results show that the same regions also exhibit increased functional connectivity in response to surprise and uncertainty. Our previous report also identified a subset of these regions, including parts of the dorsomedial frontal cortex, anterior insula, inferior frontal cortex, posterior cingulate cortex, and posterior parietal cortex, where activity was modulated by both normative (surprise and uncertainty) and non-normative (reward) factors. This smaller subset includes regions that participate in the fronto-parietal task-control, memory-retrieval, salience, and dorsal-attention systems, which is where we identified learning-related increases in functional connectivity in the current study.

Previously, we also reported regions whose univariate activity was modulated by either surprise or uncertainty alone. Surprise was associated selectively with activation in occipital cortex, and uncertainty was associated selectively with activation in anterior prefrontal and parietal cortex2. We similarly have reported multivariate activation patterns that were associated selectively with either surprise or uncertainty alone7. In the current study, we identified a key pattern of functional connectivity that was modulated by both surprise and uncertainty, but we did not identify any other pattern that was modulated reliably by either surprise or uncertainty alone. One possible explanation for this lack of a positive result was our need to use relatively long time windows (25 s, corresponding to 4-6 trials) in order to obtain reliable estimates of functional connectivity. These time windows likely included both the surprise elicited by change-points and the uncertainty that follows. Thus, functional connectivity related to surprise and uncertainty may have been difficult to dissociate temporally in our task and analysis design. Using a task that can temporally separate the tracking of surprise and uncertainty28 might enable the identification of distinct patterns of functional connectivity for each factor.

The pattern of whole-brain functional connectivity we identified here was also expressed across individuals in a manner that varied with the degree to which they updated their beliefs more in line with the normative model. Thus, individual differences in learning were also reflected in features of individual functional connectomes. In our previous study, we noted a relationship between individual differences in normative learning and the degree to which activity in dorsomedial frontal cortex and anterior insula was modulated by normative factors (surprise and uncertainty)2. Here we showed that normative learning was also associated with how functional connectivity was modulated dynamically by the same normative factors. These new findings add to previous work showing that brain network dynamics can reflect individual differences in learning in various domains12, 13, 15, 19, 22. Potentially, these differences in individual functional connectomes during learning could reflect individual differences in resting-state (task-independent) functional connectivity 29, which merits further study.

Functional connectivity captures many different kinds of statistical dependencies between brain regions, including those that result from task-driven co-activation. The strong association between neural activation and functional connectivity during periods of surprise and uncertainty in our results (Fig. 6), as well as previous studies in other domains13, 15, 17, 19, 21, 22, raises the possibility that the increases in functional connectivity between brain regions might have arisen because these regions became more tightly synchronized to external task events, without necessarily any increase in communication between them. To refute this possibility, we repeated our analyses on the predicted BOLD time series from univariate GLMs. These predicted time series, which contain only task-driven statistical dependencies between brain regions, could not recapitulate all of the effects that we observed in our actual BOLD time series. Specifically, we found modulations by task (e.g., the modulation of subgraph expression by surprise and residual updating) and individual differences (e.g., the relationship between individual differences in normative learning and the dynamic modulation of subgraph expression by normative factors) that were apparent only in the full, original functional connectivity matrices. Thus, these effects appear to include neural communications that do not simply reflect task-driven co-activation. Even though the changes in functional connectivity that we describe may reflect a mixture of task-driven and endogenous dynamics, the network analysis provides an important higher-level, reduced-dimensionality description of these changes.

### Relationship to previous results: fronto-parietal integration

A key feature of the brain-wide pattern of functional connectivity that we identified was connectivity involving the fronto-parietal task-control system. We characterized the complex pattern of functional connectivity in the learning-related subgraph by summarizing the connectivity according the putative functional system of each region27. Among all the functional systems, the largest proportion of connectivity in the learning-related subgraph involved the fronto-parietal system. Connectivity associated with the fronto-parietal system has been shown to increase at the beginning of learning and decrease toward the later phases of learning13, 19, 22. Our result extends this finding by showing that fronto-parietal functional connectivity is modulated dynamically in a trial-by-trial manner according to the need for new learning. That is, the pattern of functional connectivity captured by the learning-related subgraph increased after surprising task changes and then decreased gradually as more information was gained about the current state. The fronto-parietal system is thought of as a control system that is involved in flexible adjustments of behavior30, 31. In particular, connectivity between the fronto-parietal network and other systems has been shown to change in response to different task requirements32. This type of flexible control is critical for learning in a dynamic environment, a context in which people should adjust their degree of belief updating in a context-dependent manner1, 4.

Although the learning-related subgraph was also characterized by a balanced strength of within-system connectivity and between-system connectivity, the critical features that changed in response to task dynamics involved primarily between-system connectivity. This result implies that faster learning was associated with a greater degree of integration between different functional systems. Several previous studies have shown that complex cognitive tasks are associated with more integration between systems33, 34, 35, 36. For example, brain networks tend to be more segregated during sensory processing, and more integrated during complex cognitive tasks such as the n-back or gambling. Other work has shown that as a task becomes more practiced over time, the interaction between systems decreased while the connections within systems remained strong13. Here we demonstrated changes in integration on a fast time scale, as task demands varied from trial to trial. Integration between systems was greater during periods of the task when surprise or uncertainty was high, and therefore there was a need to update one’s beliefs and base them more on the current evidence than on expectations developed from past experience.

### Relationship to previous results: arousal and network dynamics

The dynamics of functional connectivity that we identified may be linked to arousal. We and others have shown previously that surprise and uncertainty induce changes in arousal, as indexed by pupil responses3, 6, 37. The increases in integration between the fronto-parietal and other systems that we observed in this study in response to surprise and uncertainty are consistent with previous studies of the relationship between arousal, measured in terms of wake/sleep, heart rate, and pupil-size metrics, and the dynamics of functional connectivity in large-scale brain networks36, 38, 39, 40, 41, 42. Specifically, high arousal states are associated with changes in connectivity in regions in the fronto-parietal, dorsal attention, ventral attention, salience, and default mode systems38, 41, and particularly to increases in integration in the fronto-parietal and default-mode systems36.

Potential links between arousal and dynamic functional connectivity are particularly interesting in light of theories about the functional role of arousal in changing neural gain, or the sensitivity of neural communication43. Arousal is associated with changes in the activity of noradrenergic neurons in the locus coeruleus44, 45, 46, which is hypothesized to modulate neural gain in their cortical projection targets. As neural gain increases, neurons receiving excitatory input become more active while neurons receiving inhibitory input become less active. These kinds of effects are thought to underlie certain changes in network topology47. One previous study used a simulated brain functional network to show that increased neural gain can lead to stronger network integration, particularly in the fronto-parietal system48. Further support for these ideas comes from studies using atomoxetine, a noradrenergic reuptake inhibitor, which when given systemically can increase network integration during performance of certain tasks49 and modulate learning in dynamic environments50. Future studies could incorporate direct measures of arousal to test these proposed relationships between arousal, network reconfiguration, and belief updating more directly.

### Conclusion

In this study, we provided a network-based perspective on the neural substrates of learning in dynamic and uncertain environments. In such environments, people should flexibly adjust between slow and fast learning: beliefs should be updated more strongly when new evidence is most informative, such as when the environment undergoes a surprising change or beliefs are highly uncertain. Here we identified a specific brain-wide pattern of functional connectivity (subgraph) that fluctuated dynamically with changes in surprise and uncertainty. The dynamics and expression of this pattern of functional connectivity also varied across individuals in a manner that reflected differences in learning. This pattern was expressed more strongly and was more strongly modulated by surprise and uncertainty in people who updated their beliefs in a more normative manner, with a stronger influence of surprise and uncertainty. The most important aspect of this learning-related pattern of functional connectivity is functional integration between the fronto-parietal and other functional systems. These results establish a novel link between dynamics adjustments in learning and dynamic, whole-brain changes in functional connectivity.

## Methods

### Participants

The dataset has been described in our previous reports2. Thirty-two individuals participated in the fMRI experiment: 17 females, mean age = 22.4 years (SD = 3.0; range 18-30). Human subject protocols were approved by the Internal Review Board in University of Pennsylvania. All participants provided informed consent before the experiment.

### Task

Each participant completed four 120-trial runs during functional magnetic resonance imaging. In each run, participants performed a predictive-inference task (Fig. 1a). On each trial, participants made a prediction about where the next bag would be dropped from an occluded helicopter by positioning a bucket along the horizontal axis (0-300) of the screen. The location of the bag was sampled from a Gaussian distribution with a mean (the location of the helicopter) and a standard deviation (noise). The standard deviation was high (SD = 25) or low (SD = 10) in different runs. The location of the helicopter usually remained stable but it changed occasionally. The probability of change was zero for the first three trials after a change and 0.125 for the following trials. When the location changed, the new location was sampled from a uniform distribution. Correctly predicting the location of the bag resulted in coins landing in the bucket. These coins either had positive or neutral value depending on their color, which was randomly assigned for each trial.

### Behavior model

We applied the same normative model described in our previous study2. An approximation to the ideal observer solution to this task updates beliefs according to a delta learning rule (Fig. 1b): ![Formula][2]</img>  ![Formula][3]</img>  where δt is the prediction error, which is the difference between the observed outcome (bag drop location, *X*t) and the prediction (bucket location, *B*t). Beliefs are updated in proportion to the prediction error, and this proportion is determined by αt, the learning rate. The learning rate is adjusted adaptively on each trial according to two normative factors (Fig. 1c): ![Formula][4]</img>  where Ω*t* is the change-point probability (CPP) and *τ**t* is the relative uncertainty (RU). CPP reflects the likelihood that a change-point has happened: ![Formula][5]</img>  where *U*(*X**t*|0, 300) indicates the probability of *X**t* from a uniform distribution between 0 and 300, ![Graphic][6]</img> indicates the probability of *X**t* from a Gaussian distribution with mean of *B**t* and variance of ![Graphic][7]</img> is the variance of predictive distribution of the bag location, and *H* is the average probability of change (0.1) across trials.

RU reflects the uncertainty about the current location of the helicopter relative to the amount of noise in the environment: ![Formula][8]</img>  where ![Graphic][9]</img> is the variance of outcome distribution used to generate the location of bag. There are three terms present in both the numerator and denominator. The first term is the variance of the helicopter distribution conditional on a change-point while the second term is the variance of the helicopter distribution conditional on no change-point. The third term reflects the variance due to the difference in mean between the two conditional distributions. The three terms together capture the uncertainty about the location of the helicopter.

Figure 1c shows an example of the dynamics of CPP and RU. CPP increases when there is an unexpectedly large prediction error. RU increases after CPP increases and decays slowly as more precise estimates of the helicopter location are possible.

As in our previous study, a regression model was applied to investigate how the factors in this normative model, as well as other aspects of the task, influenced participants’ belief updates. We regressed trial-by-trial updates (*B**t*+1 − *B**t*) against the prediction error (δ*t*), the interaction between prediction error and the two factors from the normative model, CPP (Ω*t*) and RU (*τ**t*), as well as the interaction between prediction error and whether the outcome was rewarded or not2. Note that these interaction terms test whether learning rates vary as a function of CPP, RU or reward; if these coefficients are zero, updates would be a constant fraction of prediction errors (given by the β$term). The form of the regression model can be written as ![Formula][10]</img>  where *Edge* is regressor of no interest that captures the tendency to avoid updating toward the edges of the screen ((150 − *B**t*+1)|150 − *B**t*+1|). This regression model was fitted separately to each participant’s data to estimate the influence of each factor on each participant’s behavior. We used the residuals of this regression to examine the relationship between subgraph expression and residual updating. To examine the relationship between individual differences in normative learning and functional network dynamics, we used the sum of the regression coefficients on the CPP term (*β*3) and the RU term (*β*5) as an index of normative learning.

### MRI data acquisition and preprocessing

MRI data were collected on a 3T Siemens Trio with a 32-channel head coil. Functional data were acquired using gradient-echo echoplanar imaging (EPI) (3 mm isotropic voxels, 64 x 64 matrix, 42 axial slices tilted 30° from the AC-PC plane, TE = 25 ms, flip angle = 75°, TR = 2500 ms). There were four runs with 226 images per run. T1-weighted MPRAGE structural images (0.9375 × 0.9375 × 1 mm voxels, 192 × 256 matrix, 160 axial slices, TI = 1100 ms, TE = 3.11 ms, flip angle = 15°, TR = 1630 ms) and matched fieldmap images (TE = 2.69 and 5.27 ms, flip angle = 60°, TR = 1000 ms) were also collected. Data were preprocessed with FSL51, 52 and AFNI53, 54. Functional data were corrected for slice timing (AFNI’s 3dTshift) and head motion (FSL’s MCFLIRT), attenuated for outliers (AFNI’s 3dDespike), undistorted and warped to MNI space (FSL’s FLIRT and FNIRT), smoothed with 6 mm FWHM Gaussian kernel (FSL’s fslmaths) and intensity-scaled by the grand-mean value per run. Details of these preprocessing steps were described in our previous study2. Structural images were segmented into gray matter, white matter (WM) and cerebrospinal fluid (CSF) (FSL’s FAST)55.

### Constructing time-varying functional networks

For each run and each participant, blood-oxygenation-level-dependent (BOLD) time series were obtained from each of 264 regions of interest (ROIs; diameter = 9mm) based on the previously defined parcellation27. ROIs that did not have valid BOLD time series for all runs and all participants were removed, resulting in *N* = 247 ROIs. For each BOLD time series, a band-pass filter was applied with a cutoff of 0.01-0.08 Hz. This low-frequency band has been shown to reflect neuronal activation and neural synchronization56, 57, 58. To remove the influence of head motion, a confound regression was implemented to regress out nuisance factors from each BOLD time series. This confound regression included 24 motion parameters (three translation and three rotation motion parameters and their expansion ![Graphic][11]</img>)59, as well as average signals from WM and CSF60.

In order to construct dynamic functional networks, we defined sliding time windows and calculated Pearson correlation coefficients between ROI time series in each sliding time window. We assigned these coefficients to the first TR in the time windows. To ensure magnetization equilibrium, the first 6 volumes of each run were removed from the analysis. For the rest of the volumes in each run, a sliding window was defined with a 10-TR (25 seconds) length and 80% overlap across windows. Each run had 106 sliding time windows, leading to *T* = 424 sliding time windows for each participant. Each participant’s data thus formed a matrix of dynamic functional networks with dimensions *N*×*N*×*N*. Then, we took each participant’s *N*×*N* matrix and unfurled the upper triangle into an ![Graphic][12]</img> vector. By concatenating vectors across all time windows (*T*), we obtained an ![Graphic][13]</img> matrix. Furthermore, we concatenated matrices from *S* = 32 participants to form a ![Graphic][14]</img> matrix. To ensure that our approach did not give undue preference to either positively or negatively weighted functional edges, we separated this matrix into two thresholded matrices: one composed of positively weighted edges, and one composed of negatively weighted edges. That is, in the matrix of positive functional correlations between ROI time series, the original negative correlations between ROI time series were set to 0; in the matrix of negative functional correlations between ROI time series, all values were multiplied by −1, and the original positive functional correlations between ROI time series were set to 0. After concatenating the matrix composed of positively weighted edges and the matrix of negatively weighted edges, we had a final ![Graphic][15]</img> matrix **A**.

### Clustering functional networks into subgraphs

We applied an unsupervised machine learning algorithm called non-negative matrix factorization (NMF)23 on ***A*** to identify subgraphs ***W*** and the time-dependent expressions of subgraphs ***H***. The matrix factorization problem **A** ≈ ***WH*** s. t. ***W*** ≥ 0, ***H*** ≥ 0 was solved by optimization of the cost function: ![Formula][16]</img>  where ***A*** is the functional connectivity matrix, ***W*** is a matrix of subgraph connectivity with size ![Graphic][17]</img> and ***H*** is a matrix of time-dependent expression coefficients for subgraphs with size *K*×(*T*×*S*×2). The parameter *k* is the number of subgraphs, α is a regularization of the connectivity for subgraphs, and β is a penalty that imposes sparsity on the temporal expression coefficients61. For fast and efficient factorization to solve this equation, we used an alternative non-negative least square with block-pivoting method with 100 iterations62. The matrices ***W*** and ***H*** were initialized with randomized values from a uniform distribution between 0 and 1.

A random sampling procedure was used to find the optimal parameters *k*, α, and β63. In this procedure, the NMF algorithm was re-run 1,000 times with parameter *k* drawn from *U*(2,15), parameter α drawn from *U*(0.01, 1), and parameter β drawn from *U*(0.01, 1). The subgraph learning performance was evaluated through 4-fold cross-validation. In each fold, twenty-four participants were used for training; Eight participants were used for testing and calculating cross-validation error ![Graphic][18]</img>. An optimal parameter set should minimize the cross-validation error. We chose an optimal parameter set (*K* = 10, α = 0.535, β = 0.230) that ensured the cross-validation error in the bottom 25% of the distribution of cross-validation error from our random sampling scheme25.

Since the result of NMF is non-deterministic, we implemented consensus clustering to obtain reliable subgraphs64. In this procedure, we (i) used the optimal parameters and ran the NMF 100 times on ***A***, (ii) concatenated subgraph matrix ***W*** across 100 runs into an aggregate matrix with dimensions ![Graphic][19]</img>, (iii) applied NMF to this aggregate matrix to obtain a final set of subgraphs ***W**consensus* and expression coefficients ***H**consensus*.

### Properties of subgraphs

Applying NMF yielded a set of subgraphs, or patterns of functional connectivity (***W***), and the expression of these subgraphs over time (***H***). To understand the subgraphs, we first rearranged ***W*** into *k* different *N*×*N* subgraphs. To understand the roles of cognitive systems in each subgraph, we mapped each ROI to 13 putative cognitive systems from the previously defined parcellation: uncertain, sensory, cingulo-opercular task control, auditory, default mode, memory retrieval, visual, fronto-parietal task control, salience, subcortical, dorsal attention, ventral attention, and cerebellar24, 27. This yielded a 13 × 13 representation of each subgraph. To show which within-system and between-system edges in this representation were strongest, we applied a permutation test. We permuted the system label for ROIs and formed a matrix with system-by-system edges. This process was repeated 10,000 times to determine which strength of system-by-system edges was above the 95% confidence interval threshold after correction for multiple comparisons.

To characterize the connectivity pattern of each subgraph, we ordered them according to the relative strength of within-system edges versus between-system edges. For each subgraph, we calculated the average strength of within-system edges (edges that link two ROIs that both belong to the same system), and the average strength of between-system edges (edges that link an ROI in one system to an ROI in another system). Then, we subtracted the average strength of between-system edges (*EB*) from the average strength of within-system edges (*EW*) and divided this difference by the sum of them ![Graphic][20]</img>. We estimated the 95% confidence interval of these measures (average relative strength, average within-system strength or average between-system strength) by implementing bootstrapping 10,000 times.

Next, we investigated the relationship between these connectivity patterns and the temporal expression of each subgraph. As the matrix of functional connectivity was divided in two, with the first half reflecting positive connectivity and the second half reflecting negative connectivity, the temporal expression matrix also had two halves, with the first reflecting positive expression over time and the second reflecting negative expression over time. As there was a strong negative correlation between positive and negative expression, we did all of our analyses on the relative expression (positive expression minus negative expression) of each subgraph26. Across subgraphs, we calculated Pearson correlation coefficients between the average relative expression and the average within-system strength, average between-system strength, and average relative strength of each subgraph. To determine the significance of the correlation coefficients, we implemented 10,000 permutations of the subgraph labels to form the null distribution of correlation coefficients.

### Modulation of subgraph expression by task factors

We investigated how fluctuations in the trial-by-trial relative expression of each subgraph were related to four trial-by-trial task factors: CPP, RU, reward, and residual updating. CPP and RU were estimated based on the normative learning model1, 2, 3. Residual updating was derived as the residual of the behavioral regression model described above. We examined the effect of these four trial-by-trial task factors together, including all four in a regression model predicting trial-by-trial relative expression. Since NMF yielded values of temporal expression every 2 TRs (5 seconds), we applied a linear interpolation on the temporal expression values to obtain an expression value aligned with outcome onset on each trial. Regression models were implemented for each participant separately. Regression coefficients were then tested at the group level using two-tailed *t*-tests.

### The relationship between subgraph expression and individual differences in normative learning

Next, we examined the relationship between subgraph expression and individual differences in the extent to which belief updating followed normative principles. As an index of normative learning for each individual, we used the sum of the regression coefficients on the CPP term (*β3*) and the RU term (*β5*) in the behavior model2. This normative learning index reflected the extent to which a participant’s trial-by-trial updates were influenced by the two normative factors CPP and RU. We examined the relationship between this index and two aspects of subgraph expression. First, across subjects, we calculated the Pearson correlation coefficient between normative learning and the dynamic modulation of relative expression by normative factors for each subgraph. This dynamic modulation was indexed as the sum of the regression coefficients for CPP and RU from the regression model predicting trial-by-trial relative expression. That is, dynamic modulation reflected how normative factors were associated with the change in relative expression of the subgraph. Second, across subjects, we calculated the Pearson correlation coefficient between normative learning and the average relative expression of each subgraph. To determine the significance of these correlation coefficients, we permuted the participant labels 10,000 times to form the null distribution.

### The relationship between task and individual difference effects across subgraphs

To show the relative importance of each subgraph, we examined the variance explained for task effects and individual differences effects for each subgraph. The variance explained for task effects was the *R2* of the regression model that investigated the relationship between the four task factors (CPP, RU, reward and residual updating) and trial-by-trial subgraph relative expression. The variance explained for individual differences effects was the *R2* of a regression model that investigated the relationship between individual differences in normative learning and two individual-level measures of subgraph relative expression (dynamic modulation by normative learning factors and average relative expression). We investigated the relationship between variance explained for task and individual differences effects using Pearson correlation. To determine the significance of this correlation coefficient, we permuted the subgraph labels 10,000 times to form the null distribution of correlation coefficients.

### Contributions of different types of edges to task and individual differences effects

We evaluated the contributions of different types of edges to the task effects (influence of CPP, RU, reward and residual updating on subgraph expression across time) and individual differences effects (relationship between normative learning and subgraph expression across subjects). We mainly focused on the contribution of within-system edges and between-system edges. For this analysis, we implemented three types of comparison: Within versus All, Between versus All, and Between versus Within. For Within versus All, we kept within-system edges only and re-estimated task and individual differences effects; then, we compared these effects with the effects estimated using all edges. This comparison showed the change of effects after between-system edges were removed, and thus, this comparison revealed the contribution of between-system edges. For Between versus All, we kept between-system edges only and re-estimated task and individual differences effects. We then compared these effects with the effects estimated using all edges. In this comparison, within-system edges were removed and thus, we examined the contribution of within-system edges. Last, the comparison of Between versus Within is a direct comparison between effects estimated with between-system edges only and effects estimated with within-system edges only. Thus, this comparison examined the different contributions of between-system and within-system edges.

Specifically, for task effects, we examined the change of coefficients in the regression model that investigated the influence of four task factors—CPP, RU, reward and residual updating—on subgraph relative expression. The change was calculated for each participant separately, and the significance of change was then tested at the group level using two-tailed *t*-tests. For individual differences effects, we examined the change of correlation coefficients for two types of relationship: the relationship between individual normative learning and dynamic modulation of subgraph relative expression and the relationship between individual normative learning and average subgraph relative expression. To determine the significance of the change of correlation coefficients, we permuted the labels of participants for individual normative learning 10,000 times to form the null distribution of the change of correlation coefficients.

We also investigated the contribution of different functional systems and the contribution of different system-by-system edges. For the contribution of different functional systems, we compared the effects after removing edges of one functional system with the effects estimated with all edges. For the contribution of different system-by-system edges, we compared the effects after removing one system-by-system edge with the effects estimated with all edges. Statistical testing was conducted with the same procedures described in the previous paragraph.

### Relationship between univariate activation and functional connectivity

To investigate the relationship between dynamic functional connectivity and univariate activation, we fit a mass univariate GLM. In this GLM, the regressors were the outcome onset and four modulators of outcome onset: CPP, RU, reward and residual updating. These regressors were convolved with a gamma hemodynamic response function (HRF) as well as the temporal derivative of this function. Six motion parameters were also included as regressors.

To examine what aspects of our functional connectivity results could be accounted for by functional coactivation, we used the regression coefficients from the GLM above (including both the main HRF and its temporal derivative for each regressor) to create a predicted BOLD time series. We then repeated the same sequence of analyses described above on this predicted BOLD time series. This predicted BOLD time series reflected the modulation of univariate activation by CPP, RU, reward and residual updating. However, this predicted BOLD time series lacked any statistical dependencies between regions that were present in the actual BOLD time series that could not be explained by task-driven changes in univariate activation. Thus, any functional connectivity results we observed with this predicted BOLD time series could be fully accounted for by task-driven changes in univariate activation.

## Supporting information

Supplementary figures [[supplements/800284_file03.pdf]](pending:yes)

## Data availability

The data for the current study are available from the corresponding author upon request.

## Code availability

Code is available at [https://github.com/changhaokao/nmf\_network\_learning](https://github.com/changhaokao/nmf_network_learning).

## Author contributions

C.-H.K. and J.W.K. designed the study. A.N.K. established the software for NMF. A.N.K. and D.S.B. provided suggestions in network analyses. C.-H.K. implemented all the analyses and visualization and drafted the manuscript. All authors interpreted the results and revised the manuscript.

## Competing interests

The authors declare no competing interests.

## Acknowledgements

This work was supported by grants from National Institute of Mental Health (R01-MH098899 to J.I.G. and J.W.K.) and National Science Foundation (1533623 to J.I.G. and J.W.K.). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

*   Received October 9, 2019.
*   Revision received January 16, 2020.
*   Accepted January 17, 2020.


*   © 2020, Posted by Cold Spring Harbor Laboratory

This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/)

## References

1.  1.Nassar MR, Wilson RC, Heasly B, Gold JI. An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. The Journal of Neuroscience 30, 12366–12378 (2010).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Njoiam5ldXJvIjtzOjU6InJlc2lkIjtzOjExOiIzMC8zNy8xMjM2NiI7czo0OiJhdG9tIjtzOjM3OiIvYmlvcnhpdi9lYXJseS8yMDIwLzAxLzE3LzgwMDI4NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

2.  2.McGuire JT, Nassar MR, Gold JI, Kable JW. Functionally Dissociable Influences on Learning Rate in a Dynamic Environment. Neuron 84, 870–881 (2014).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuron.2014.10.013&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=25459409&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 

3.  3.Nassar MR, Rumsey KM, Wilson RC, Parikh K, Heasly B, Gold JI. Rational regulation of learning dynamics by pupil-linked arousal systems. Nature Neuroscience 15, 1040–1046 (2012).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/nn.3130&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=22660479&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 

4.  4.Behrens TEJ, Woolrich MW, Walton ME, Rushworth MFS. Learning the value of information in an uncertain world. Nature Neuroscience 10, 1214–1221 (2007).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/nn1954&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=17676057&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000249144000025&link_type=ISI) 

5.  5.Meder D, et al. Simultaneous representation of a spectrum of dynamically changing value estimates during decision making. Nature Communications 8, 1942 (2017).
    
    

6.  6.O’Reilly JX, Schüffelgen U, Cuell SF, Behrens TEJ, Mars RB, Rushworth MFS. Dissociable effects of surprise and model update in parietal and anterior cingulate cortex. Proceedings of the National Academy of Sciences 110, E3660–E3669 (2013).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMjoiMTEwLzM4L0UzNjYwIjtzOjQ6ImF0b20iO3M6Mzc6Ii9iaW9yeGl2L2Vhcmx5LzIwMjAvMDEvMTcvODAwMjg0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

7.  7.Nassar MR, McGuire JT, Ritz H, Kable JW. Dissociable Forms of Uncertainty-Driven Representational Change Across the Human Brain. The Journal of Neuroscience 39, 1688–1698 (2019).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Njoiam5ldXJvIjtzOjU6InJlc2lkIjtzOjk6IjM5LzkvMTY4OCI7czo0OiJhdG9tIjtzOjM3OiIvYmlvcnhpdi9lYXJseS8yMDIwLzAxLzE3LzgwMDI4NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

8.  8.Friston KJ. Functional and Effective Connectivity: A Review. Brain Connectivity 1, 13–36 (2011).
    
    

9.  9.Bassett DS, Sporns O. Network neuroscience. Nature Neuroscience 20, 353–364 (2017).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/nn.4502&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=28230844&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 

10. 10.Bullmore E, Sporns O. Complex brain networks: graph theoretical analysis of structural and functional systems. Nature Reviews Neuroscience 10, 186 (2009).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/nrn2575&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=19190637&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000263556500012&link_type=ISI) 

11. 11.Medaglia JD, Lynall M-E, Bassett DS. Cognitive network neuroscience. Journal of cognitive neuroscience 27, 1471–1491 (2015).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access\_num=10.1162/jocn_a_00810&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=25803596&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 

12. 12.Bassett DS, Wymbs NF, Porter MA, Mucha PJ, Carlson JM, Grafton ST. Dynamic reconfiguration of human brain networks during learning. Proceedings of the National Academy of Sciences 108, 7641–7646 (2011).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMToiMTA4LzE4Lzc2NDEiO3M6NDoiYXRvbSI7czozNzoiL2Jpb3J4aXYvZWFybHkvMjAyMC8wMS8xNy84MDAyODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

13. 13.Bassett DS, Yang M, Wymbs NF, Grafton ST. Learning-induced autonomy of sensorimotor systems. Nature Neuroscience 18, 744–751 (2015).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/nn.3993&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=25849989&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 

14. 14.Mattar MG, Wymbs NF, Bock AS, Aguirre GK, Grafton ST, Bassett DS. Predicting future learning from baseline network architecture. NeuroImage 172, 107–117 (2018).
    
    

15. 15.Gerraty RT, Davidow JY, Foerde K, Galvan A, Bassett DS, Shohamy D. Dynamic Flexibility in Striatal-Cortical Circuits Supports Reinforcement Learning. The Journal of Neuroscience 38, 2442–2453 (2018).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Njoiam5ldXJvIjtzOjU6InJlc2lkIjtzOjEwOiIzOC8xMC8yNDQyIjtzOjQ6ImF0b20iO3M6Mzc6Ii9iaW9yeGl2L2Vhcmx5LzIwMjAvMDEvMTcvODAwMjg0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

16. 16.Mattar MG, Thompson-Schill SL, Bassett DS. The network architecture of value learning. Network Neuroscience 2, 128–149 (2018).
    
    

17. 17.Antzoulatos Evan G, Miller Earl K. Increases in Functional Connectivity between Prefrontal Cortex and Striatum during Category Learning. Neuron 83, 216–225 (2014).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuron.2014.05.005&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=24930701&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 

18. 18.Baeg EH, Kim YB, Kim J, Ghim J-W, Kim JJ, Jung MW. Learning-Induced Enduring Changes in Functional Connectivity among Prefrontal Cortical Neurons. The Journal of Neuroscience 27, 909–918 (2007).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Njoiam5ldXJvIjtzOjU6InJlc2lkIjtzOjg6IjI3LzQvOTA5IjtzOjQ6ImF0b20iO3M6Mzc6Ii9iaW9yeGl2L2Vhcmx5LzIwMjAvMDEvMTcvODAwMjg0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

19. 19.Büchel C, Coull JT, Friston KJ. The predictive value of changes in effective connectivity for human learning. Science 283, 1538–1541 (1999).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoic2NpIjtzOjU6InJlc2lkIjtzOjEzOiIyODMvNTQwNy8xNTM4IjtzOjQ6ImF0b20iO3M6Mzc6Ii9iaW9yeGl2L2Vhcmx5LzIwMjAvMDEvMTcvODAwMjg0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

20. 20.Fatima Z, Kovacevic N, Misic B, McIntosh AR. Dynamic functional connectivity shapes individual differences in associative learning. Human Brain Mapping 37, 3911–3928 (2016).
    
    

21. 21.Lewis CM, Baldassarre A, Committeri G, Romani GL, Corbetta M. Learning sculpts the spontaneous activity of the resting human brain. Proceedings of the National Academy of Sciences 106, 17558–17563 (2009).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMjoiMTA2LzQxLzE3NTU4IjtzOjQ6ImF0b20iO3M6Mzc6Ii9iaW9yeGl2L2Vhcmx5LzIwMjAvMDEvMTcvODAwMjg0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

22. 22.Sun FT, Miller LM, Rao AA, D’esposito M. Functional connectivity of cortical networks involved in bimanual motor sequence learning. Cerebral cortex 17, 1227–1234 (2006).
    
    

23. 23.Lee DD, Seung HS. Learning the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/44565&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=10548103&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000083368700052&link_type=ISI) 

24. 24.Chai LR, et al. Evolution of brain network dynamics in neurodevelopment. Network Neuroscience 1, 14–30 (2017).
    
    

25. 25.Khambhati AN, Mattar MG, Wymbs NF, Grafton ST, Bassett DS. Beyond modularity: Fine-scale mechanisms and rules for brain network reconfiguration. NeuroImage 166, 385–399 (2018).
    
    

26. 26.Khambhati AN, Medaglia JD, Karuza EA, Thompson-Schill SL, Bassett DS. Subgraphs of functional brain networks identify dynamical constraints of cognitive control. PLOS Computational Biology 14, e1006234 (2018).
    
    

27. 27.Power JD, et al. Functional Network Organization of the Human Brain. Neuron 72, 665–678 (2011).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuron.2011.09.006&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=22099467&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000297180100016&link_type=ISI) 

28. 28.Nassar MR, Bruckner R, Gold JI, Li S-C, Heekeren HR, Eppinger B. Age differences in learning emerge from an insufficient representation of uncertainty in older adults. Nature communications 7, 11609 (2016).
    
    

29. 29.Tompson S, Falk EB, Vettel JM, Bassett DS. Network Approaches to Understand Individual Differences in Brain Connectivity: Opportunities for Personality Neuroscience. Personality Neuroscience 1, e5 (2018).
    
    

30. 30.Cole MW, Repovš G, Anticevic A. The Frontoparietal Control System:A Central Role in Mental Health. The Neuroscientist 20, 652–664 (2014).
    
    

31. 31.Power JD, Petersen SE. Control-related systems in the human brain. Current Opinion in Neurobiology 23, 223–228 (2013).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.conb.2012.12.009&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=23347645&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 

32. 32.Cole MW, Reynolds JR, Power JD, Repovs G, Anticevic A, Braver TS. Multi-task connectivity reveals flexible hubs for adaptive task control. Nature Neuroscience 16, 1348–1355 (2013).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/nn.3470&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=23892552&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 

33. 33.Bertolero MA, Yeo BTT, D’Esposito M. The modular and integrative functional architecture of the human brain. Proceedings of the National Academy of Sciences 112, E6798–E6807 (2015).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMjoiMTEyLzQ5L0U2Nzk4IjtzOjQ6ImF0b20iO3M6Mzc6Ii9iaW9yeGl2L2Vhcmx5LzIwMjAvMDEvMTcvODAwMjg0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

34. 34.Shine JM, Poldrack RA. Principles of dynamic network reconfiguration across diverse brain states. NeuroImage 180, 396–405 (2017).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2017.08.010&link_type=DOI) 

35. 35.Cohen JR, D’Esposito M. The Segregation and Integration of Distinct Brain Networks and Their Relationship to Cognition. The Journal of Neuroscience 36, 12083–12094 (2016).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Njoiam5ldXJvIjtzOjU6InJlc2lkIjtzOjExOiIzNi80OC8xMjA4MyI7czo0OiJhdG9tIjtzOjM3OiIvYmlvcnhpdi9lYXJseS8yMDIwLzAxLzE3LzgwMDI4NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

36. 36.Shine JM, et al. The Dynamics of Functional Brain Networks: Integrated Network States during Cognitive Task Performance. Neuron 92, 544–554 (2016).
    
    

37. 37.Krishnamurthy K, Nassar MR, Sarode S, Gold JI. Arousal-related adjustments of perceptual biases optimize perception in dynamic environments. Nature Human Behaviour 1, 0107 (2017).
    
    

38. 38.Wang C, Ong JL, Patanaik A, Zhou J, Chee MWL. Spontaneous eyelid closures link vigilance fluctuation with fMRI dynamic connectivity states. Proceedings of the National Academy of Sciences 113, 9653–9658 (2016).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMToiMTEzLzM0Lzk2NTMiO3M6NDoiYXRvbSI7czozNzoiL2Jpb3J4aXYvZWFybHkvMjAyMC8wMS8xNy84MDAyODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

39. 39.Chang C, et al. Tracking brain arousal fluctuations with fMRI. Proceedings of the National Academy of Sciences 113, 4518–4523 (2016).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMToiMTEzLzE2LzQ1MTgiO3M6NDoiYXRvbSI7czozNzoiL2Jpb3J4aXYvZWFybHkvMjAyMC8wMS8xNy84MDAyODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

40. 40.Haimovici A, Tagliazucchi E, Balenzuela P, Laufs H. On wakefulness fluctuations as a source of BOLD functional connectivity dynamics. Scientific Reports 7, 5908 (2017).
    
    

41. 41.Patanaik A, Tandi J, Ong JL, Wang C, Zhou J, Chee MWL. Dynamic functional connectivity and its behavioral correlates beyond vigilance. NeuroImage 177, 1–10 (2018).
    
    

42. 42.Young CB, et al. Dynamic Shifts in Large-Scale Brain Network Balance As a Function of Arousal. The Journal of Neuroscience 37, 281–290 (2017).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Njoiam5ldXJvIjtzOjU6InJlc2lkIjtzOjg6IjM3LzIvMjgxIjtzOjQ6ImF0b20iO3M6Mzc6Ii9iaW9yeGl2L2Vhcmx5LzIwMjAvMDEvMTcvODAwMjg0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

43. 43.Servan-Schreiber D, Printz H, Cohen JD. A network model of catecholamine effects: gain, signal-to-noise ratio, and behavior. Science 249, 892–895 (1990).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoic2NpIjtzOjU6InJlc2lkIjtzOjEyOiIyNDkvNDk3MS84OTIiO3M6NDoiYXRvbSI7czozNzoiL2Jpb3J4aXYvZWFybHkvMjAyMC8wMS8xNy84MDAyODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

44. 44.Aston-Jones G, Cohen JD. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annual Review of Neuroscience 28, 403–450 (2005).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1146/annurev.neuro.28.061604.135709&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=16022602&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000231235700016&link_type=ISI) 

45. 45.Sara Susan J. The locus coeruleus and noradrenergic modulation of cognition. Nature Reviews Neuroscience 10, 211–223 (2009).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/nrn2573&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=19190638&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000263556500014&link_type=ISI) 

46. 46.Sara Susan J, Bouret S. Orienting and Reorienting: The Locus Coeruleus Mediates Cognition through Arousal. Neuron 76, 130–141 (2012).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuron.2012.09.011&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=23040811&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000309799000010&link_type=ISI) 

47. 47.Eldar E, Cohen JD, Niv Y. The effects of neural gain on attention and learning. Nature Neuroscience 16, 1146–1153 (2013).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1038/nn.3428&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=23770566&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 

48. 48.Shine JM, Aburn MJ, Breakspear M, Poldrack RA. The modulation of neural gain facilitates a transition between functional segregation and integration in the brain. Elife 7, e31130 (2018).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.7554/eLife.31130&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=29376825&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 

49. 49.Shine JM, van den Brink RL, Hernaus D, Nieuwenhuis S, Poldrack RA. Catecholaminergic manipulation alters dynamic network topology across cognitive states. Network Neuroscience 2, 381–396 (2018).
    
    

50. 50.Jepma M, Murphy PR, Nassar MR, Rangel-Gomez M, Meeter M, Nieuwenhuis S. Catecholaminergic Regulation of Learning Rate in a Dynamic Environment. PLOS Computational Biology 12, e1005171 (2016).
    
    

51. 51.Jenkinson M, Beckmann CF, Behrens TEJ, Woolrich MW, Smith SM. FSL. NeuroImage 62, 782–790 (2012).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2011.09.015&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=21979382&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000306390600032&link_type=ISI) 

52. 52.Smith SM, et al. Advances in functional and structural MR image analysis and implementation as FSL. NeuroImage 23, S208–S219 (2004).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2004.07.051&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=15501092&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000225374100020&link_type=ISI) 

53. 53.Cox RW. AFNI: Software for Analysis and Visualization of Functional Magnetic Resonance Neuroimages. Computers and Biomedical Research 29, 162–173 (1996).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1006/cbmr.1996.0014&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=8812068&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=A1996UV56700002&link_type=ISI) 

54. 54.Cox RW. AFNI: What a long strange trip it’s been. NeuroImage 62, 743–747 (2012).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2011.08.056&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=21889996&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000306390600026&link_type=ISI) 

55. 55.Zhang Y, Brady M, Smith S. Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm. IEEE Transactions on Medical Imaging 20, 45–57 (2001).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1109/42.906424&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=11293691&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000167324900005&link_type=ISI) 

56. 56.Biswal B, Zerrin Yetkin F, Haughton VM, Hyde JS. Functional connectivity in the motor cortex of resting human brain using echo-planar mri. Magnetic Resonance in Medicine 34, 537–541 (1995).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1002/mrm.1910340409&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=8524021&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=A1995RX75100008&link_type=ISI) 

57. 57.Lu H, et al. Synchronized delta oscillations correlate with the resting-state functional MRI signal. Proceedings of the National Academy of Sciences 104, 18265–18269 (2007).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMjoiMTA0LzQ2LzE4MjY1IjtzOjQ6ImF0b20iO3M6Mzc6Ii9iaW9yeGl2L2Vhcmx5LzIwMjAvMDEvMTcvODAwMjg0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

58. 58.Zuo X-N, et al. The oscillating brain: Complex and reliable. NeuroImage 49, 1432–1445 (2010).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2009.09.037&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=19782143&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=000272808400028&link_type=ISI) 

59. 59.Friston KJ, Williams S, Howard R, Frackowiak RSJ, Turner R. Movement-Related effects in fMRI time-series. Magnetic Resonance in Medicine 35, 346–355 (1996).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1002/mrm.1910350312&link_type=DOI) 
    
    [PubMed](http://biorxiv.org/lookup/external-ref?access_num=8699946&link_type=MED&atom=%2Fbiorxiv%2Fearly%2F2020%2F01%2F17%2F800284.atom) 
    
    [Web of Science](http://biorxiv.org/lookup/external-ref?access_num=A1996TX77500009&link_type=ISI) 

60. 60.Fox MD, Snyder AZ, Vincent JL, Corbetta M, Van Essen DC, Raichle ME. The human brain is intrinsically organized into dynamic, anticorrelated functional networks. Proceedings of the National Academy of Sciences 102, 9673–9678 (2005).
    
    [Abstract/FREE Full Text](http://biorxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoicG5hcyI7czo1OiJyZXNpZCI7czoxMToiMTAyLzI3Lzk2NzMiO3M6NDoiYXRvbSI7czozNzoiL2Jpb3J4aXYvZWFybHkvMjAyMC8wMS8xNy84MDAyODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

61. 61.Kim J, Park H. Fast Nonnegative Matrix Factorization: An Active-Set-Like Method and Comparisons. SIAM Journal on Scientific Computing 33, 3261–3281 (2011).
    
    

62. 62.Kim J, He Y, Park H. Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework. Journal of Global Optimization 58, 285–319 (2014).
    
    

63. 63.Bergstra J, Bengio Y. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 281–305 (2012).
    
    [CrossRef](http://biorxiv.org/lookup/external-ref?access_num=10.1162/153244303322533223&link_type=DOI) 

64. 64.Monti S, Tamayo P, Mesirov J, Golub T. Consensus Clustering: A Resampling-Based Method for Class Discovery and Visualization of Gene Expression Microarray Data. Machine Learning 52, 91–118 (2003).

 [1]: /embed/inline-graphic-1.gif
 [2]: /embed/graphic-7.gif
 [3]: /embed/graphic-8.gif
 [4]: /embed/graphic-9.gif
 [5]: /embed/graphic-10.gif
 [6]: /embed/inline-graphic-2.gif
 [7]: /embed/inline-graphic-3.gif
 [8]: /embed/graphic-11.gif
 [9]: /embed/inline-graphic-4.gif
 [10]: /embed/graphic-12.gif
 [11]: /embed/inline-graphic-5.gif
 [12]: /embed/inline-graphic-6.gif
 [13]: /embed/inline-graphic-7.gif
 [14]: /embed/inline-graphic-8.gif
 [15]: /embed/inline-graphic-9.gif
 [16]: /embed/graphic-13.gif
 [17]: /embed/inline-graphic-10.gif
 [18]: /embed/inline-graphic-11.gif
 [19]: /embed/inline-graphic-12.gif
 [20]: /embed/inline-graphic-13.gif