Abstract
Relating neural activity to behavior requires an understanding of how neural computations arise from the coordinated dynamics of distributed, recurrently connected neural populations. However, inferring the nature of recurrent dynamics from partial recordings of a neural circuit presents significant challenges. Here, we show that some of these challenges can be overcome by a fine-grained analysis of the dynamics of neural residuals, i.e. trial-by-trial variability around the mean neural population trajectory for a given task condition. Residual dynamics in macaque pre-frontal cortex (PFC) in a saccade-based perceptual decision-making task reveals recurrent dynamics that is time-dependent, but consistently stable, and implies that pronounced rotational structure in PFC trajectories during saccades are driven by inputs from upstream areas. The properties of residual dynamics restrict the possible contributions of PFC to decision-making and saccade generation, and suggest a path towards fully characterizing distributed neural computations with large-scale neural recordings and targeted causal perturbations.
Introduction
Perception, decisions, and the resulting actions reflect neural computations implemented by large, interacting neuronal populations acting in concert. Inferring the nature of these interactions from recordings of neural activity is a key step towards uncovering the neural computations underlying behavior1–4. One promising approach is based on the premise that neural computations reflect the action of a dynamical system5–7, whereby the computations implemented by a neural population emerge from the interplay between external inputs into a distributed neural population and the internal dynamics resulting from the recurrent connections between neurons. The utility of such a “computation-through-dynamics” framework hinges critically on our ability to characterize the nature of this interplay, and disentangle the individual contributions of inputs and recurrent dynamics. In practice, disentangling these two factors based on recordings of neural responses alone is challenging, as typically neither the exact properties of the inputs into a brain area, nor the nature of recurrent connectivity within and across areas, are known a priori8–11.
Here, we show that some of the challenges inherent to inferring the contribution of recurrent dynamics to neural responses can be overcome by analyzing the dynamical structure of neural population residuals, i.e. the trial-to-trial variability in neural population responses12–22. Our approach involves solving a statistical inference problem, but is ultimately based on the intuitive idea that the effect of recurrent computations can be revealed by observing how a local perturbation of the state of the neural population evolves over time11,23–27. Unlike in causal perturbation experiments, where the perturbations are generated externally28–30, we rely entirely on an analysis of recorded response residuals, which we interpret as naturally occurring perturbations within the repertoire of neural patterns produced by a recurrent neural network31,32. We term the time-varying dynamics of response residuals as “residual dynamics”, and show that in many settings it can resolve key properties of the recurrent dynamics underlying recorded neural responses. Obtaining a complete and quantitative description of residual dynamics is difficult, because neural population residuals are typically dominated by unstructured noise. To obtain reliable and unbiased estimates of residual dynamics, we thus developed novel statistical methods based on subspace identification33,34 and instrumental variable regression35.
Our findings are organized in three sections. First, we illustrate the challenges in disentangling inputs and recurrent dynamics based on the simulations of a few, simple dynamical systems (Fig. 1-2). These dynamical systems are analogous to single-area, artificial recurrent neural networks (RNN) previously proposed for explaining the network-level mechanisms underlying sensory evidence integration36–42 and movement generation in cortical areas43–47. We demonstrate that our estimates of residual dynamics can reveal the essential features of the computations implemented by these models, even when the time-course of the inputs are unknown. Second, we study neural population recordings from pre-frontal cortex (PFC) of macaque monkeys during decision-making and saccadic choices (Fig. 3-5). While neural population trajectories in PFC are consistent with a number of previously proposed models of evidence integration and movement generation, we are able to rule out several candidate models based on the properties of the inferred residual dynamics. Third, we analyze simulated responses of a previously proposed multi-area RNN model of decision-making48, to illustrate how inferred residual dynamics can be used to deduce circuit-level implementations of distributed recurrent computations. (Fig. 6-8).
Results
A prevalent approach for studying population-level neural computations relies on extracting low-dimensional neural trajectories from the population response49–55. The time-course of such trajectories and their dependency on task-variables can be compared to those generated by hand-designed41,56–60 and task-optimized RNN models42,46,47,61–66, or statistical models of neural dynamics67–75. Such an approach has been very successful in generating hypotheses about the nature of neural computations, but typically cannot unambiguously resolve the properties of recurrent dynamics based on the measured population responses10,11—estimating these properties is generally an ill-posed problem whenever other factors contributing to the responses, like external inputs, are unknown or unobserved.
Neural population trajectories poorly constrain recurrent computations
To illustrate the nature of this problem, we consider simulated responses of a number of distinct models of neural population dynamics during perceptual decision-making40 and movement generation43 (Fig. 1). While these hand-designed models are not meant to precisely reproduce neural recordings, they do capture the distinctive features of rather complex, non-linear RNNs trained to integrate sensory evidence towards a choice41,42,76 (Fig. 1a) or generate complex motor sequences46,47 (Fig. 1b).
In the models, the temporal evolution of the neural population response (zt) at time t is governed by a non-linear differential equation, which describes the momentary change in the response (zt) as resulting from the combined action of the recurrent dynamics (F), the input (ut), and the noise (ϵt): Any solution to the above equation is also determined by the initial condition z0 (the neural state at the start of the trial). Differences in responses across task-conditions (e.g., different choices or movements) are explained by allowing ut or z0 to vary across conditions (Fig. 1c, red vs. blue; Fig. 1b, initial conditions IC1 vs. IC2).
We simulated single-trial responses for two task-conditions and represented them as trajectories in a 2-dimensional neural state-space (Fig. 1a,b, choice 1 & 2; dark-gray curves). The recurrent dynamics (F) can be represented as a flow field (Fig. 1a,b, black arrows and light-gray curves), which describes how the instantaneous neural state (zt) evolves from a given location in state-space in the absence of inputs and noise. The action of the external input (ut) corresponds to injecting a pattern of activity into the neural population, and therefore pushing the trajectory along a direction in state space that can vary both across time and task conditions (Fig. 1a; red and blue arrows; Fig. 1c). For simplicity, ut only captures the component of the inputs that is deterministic, i.e. repeatable across trials of the same condition. Any trial-to-trial variability in the inputs, together with moment-to-moment variability generated intrinsically within the recurrent population, are explained by the noise ϵt and the initial-condition z0.
Critically, the simulations show that very different combinations of these factors can result in very similar trajectories. For example, the three models of decision-making differ in the nature of their inputs and recurrent dynamics, each mimicking a specific behavioral “strategy” for perceptual decision-making36–38,77–80, from unstable, impulsive decisions (Fig. 1a, saddle point), to optimal accumulation of evidence (Fig1a, line attractor), and leaky, forgetful accumulation (Fig. 1a, point attractor). Yet, for the chosen inputs, which depending on the model are either constant (Fig. 1c, saddle point) or transient (Fig. 1c, line and point attractor), all three models result in similar single-trial trajectories (Fig. 1a, gray curves) and essentially indistinguishable condition-averaged trajectories (Fig. 1a, blue and red curves).
Analogous observations hold for the models of movement generation (Fig. 1b). Two of the models have no inputs, and are driven entirely by recurrent dynamics starting from condition-dependent initial conditions—one model implements rotational dynamics45,46, implying that any variability in the initial condition on a given trial is reflected throughout the entire trajectory (Fig. 1b rotations; gray curves); the other implements what we refer to as “funnel” dynamics47,61, whereby neural activity is pushed through a narrow channel in state space, and any variability along directions orthogonal to the channel is suppressed (Fig. 1b, funnel). In the third model, the recurrent dynamics implements a point attractor, and responses are mostly input driven11 (Fig. 1b, point attractor). The simulated condition-averages can neither distinguish between the models with or without inputs, nor between the different recurrent dynamics associated with models that lack an input.
Residual dynamics as a window onto recurrent dynamics
More insights into the underlying computations can be obtained by considering the dynamics of response residuals, the component of single-trial responses that is not explained by the condition-averaged responses12,17,20,81. Residuals are defined as the difference between a given single-trial trajectory and the corresponding condition-averaged trajectory (Extended Data Fig. 1). We interpret residuals as perturbations away from the condition-averaged trajectory, and then describe how these perturbations evolve over time (Extended Data Fig. 1).
In the simulated models, the dynamics of residuals can be derived analytically (Fig. 2a, Extended Data Fig. 1). First, we define the effective dynamics, which describes how the population response would evolve from any given location in state-space and time in the absence of noise. The effective dynamics is obtained by summing the contributions of the recurrent dynamics and the input. The residual dynamics is then obtained by subtracting, from the effective dynamics, a component corresponding to the instantaneous direction of change along the condition-averaged trajectory (Fig. 2a, see labels over each panel).
The residual dynamics describes how a perturbation away from the condition-averaged neural state would evolve relative to the trajectory over the course of one time-step. In Fig. 2c,d, the blue dot indicates the unperturbed, “reference” neural state, which lies along the average trajectory. The tail of each arrow indicates the residual (the perturbed state), and the arrow-head shows how this residual evolves over one time-step. For the saddle point model (Fig. 2c, saddle point), perturbations along the horizontal direction, away from the trajectory, expand over time (arrows point away from the reference state), whereas perturbations along the vertical direction decay back to the trajectory (arrows point towards the reference state). These dynamics correctly reflect the influence of a saddle point in the vicinity of the examined region of state space (Fig. 1a, box). Likewise, the residual dynamics correctly reveals line attractor and point attractor dynamics in the other two models of decisions (Fig. 2c), as well as the main properties of the recurrent dynamics in the models of movement, i.e. rotational dynamics, decay towards the funnel, and point attractor dynamics (Fig. 2d). More generally, the residual dynamics only reflects the recurrent dynamics, rather than any external inputs, when two constraints are met. First, inputs and recurrent dynamics must combine additively. Second, the noise in the inputs (captured by ϵt in Eq. 1) must be temporally uncorrelated. Both constraints hold exactly for the models in Fig. 1, and at least approximately for many previously proposed RNN models. The second constraint, however, is likely to be violated at the level of many single areas in biological networks, as input variability may be temporally correlated when the input originates in upstream areas that themselves implement recurrent dynamics. Nonetheless, we show below that even in such scenarios residual dynamics can provide insights into the nature of recurrent dynamics in the recorded area. Unlike residual dynamics, the effective dynamics and the condition-averaged trajectories always reflect the properties of both, the recurrent dynamics and the inputs, even when the two constraints above are met.
A further, key property of residual dynamics simplifies the task of estimating it directly from neural responses, even when the underlying computations are non-linear and vary both in time and across state-space location. Residual dynamics is always expressed relative to a “reference” neural state, corresponding to a particular time and location along a condition-averaged trajectory (Fig. 2c,d, blue dot). By this definition, residual dynamics always has a fixed point at the location of the reference state (Fig. 2c,d, blue dot; see methods) making it amenable to be estimated using easily interpretable, statistical models characterized by dynamics that is linear and autonomous (i.e. without inputs). Specifically, the residual dynamics can be approximated by a condition and time-dependent, locally linear system, whereby time parameterizes location in state-space along the condition-averaged trajectory (Extended Data Fig. 1). We estimate these linear systems from neural response residuals by combining methods from subspace identification33,34 and instrumental variable regression35 (Extended Data Fig. 2). These methods, unlike simpler linear regression approaches, can produce robust and unbiased estimates of residual dynamics in biologically realistic settings (Extended Data Fig. 3).
We summarize the residual dynamics through the main properties of the estimated local linear dynamical systems, specifically the magnitude of the eigen-values (EV), the singular values (SV), and the rotation frequency associated with the EV (Fig. 2e-g). For locations close to the saddle point in the model of decision-making, one of the EV is larger than 1, implying that perturbations along the associated eigen-vector (the horizontal direction in Fig. 1a, left) expand over time; the other EV is smaller than one, corresponding to decay along the vertical direction (Fig. 1a, left; center of flow field; Fig. 2e, left-most panel; early times). A line attractor results in a single EV of 1 (Fig. 2e, second from left) as horizontal perturbations are persistent, i.e. neither expand nor decay, and a point attractor in all EV smaller than 1 (Fig. 2e, third from left; all directions decay). Rotational dynamics results in EV that are complex-valued and thus associated with a non-zero rotation frequency (Fig. 2g). Finally, differences between the magnitude of SV and EV reflect non-normal dynamics, a critical feature of a number of previous models of neural computation82–87. The SV larger than 1 in the line attractor model implies that small perturbations along the corresponding right singular vector transiently expand, even though they are persistent (EV=1) or decay (EV<1) over longer time-scales (Fig. 2e,f).
Neural population responses of decisions and movements in PFC
We compared these model dynamics to neural population responses recorded in the pre-frontal cortex (PFC; area 8Ar) of two macaque monkeys performing a saccade-based perceptual decision-making task39,81,88,89 (Fig 3a,b; Extended Data Fig. 4). To increase the statistical power of our analyses, we employed a dimensionality reduction technique to “align” the task-related subspaces of neural activity from different experiments with a similar task-configuration (Extended Data Fig. 4; 14-61 experiments per configuration; 150-200 units per experiment). This alignment yielded a shared, 20-dimensional neural state-space explaining >90% of task-related variance in the average neural responses measured across different experiments90 (Extended Data Fig. 5). All the analyses below are performed within this aligned subspace, although the main results can be reproduced from sufficiently long single experiments (Extended Data Fig. 6).
The condition-averaged population trajectories in PFC shared important features with the average trajectories of the models in Fig. 1. We visualized the population trajectories through projections onto four distinct, two-dimensional activity subspaces: a “choice” plane, emphasizing choice-related activity; a “time” plane, emphasizing time-varying activity common to both choices; and two “jPC” planes45, emphasizing rotational dynamics (Fig. 3c,d; left to right). We estimated these planes separately during a decision-epoch, which coincided with the presentation of a random-dots stimulus (Fig 3c), and during a movement-epoch aligned to the execution of the saccade (Fig. 3d). As in the decision-models (Fig. 1a), PFC responses started in an undifferentiated state prior to stimulus onset (Fig 3c; choice plane; filled dots mark stimulus onset) and gradually diverged based on the upcoming choice of the animal (Fig. 3c, red vs. blue). PFC responses during the movement period showed pronounced rotational components (Fig. 3d, jPC12 plane; filled dots mark movement onset) similar to those in the movement models (Fig. 1b). Prior to saccade-onset, PFC responses fell into largely stationary, choice-dependent states and then transitioned into rotational dynamics following the presentation of the go cue (Fig. 3d, jPC planes).
The measured PFC responses also differed from the model responses in several ways. Consistent with past reports of population dynamics during decisions, working memory and movements, PFC responses reflected strong condition-independent components during both task-epochs (e.g. Fig. 3c,d, time-plane) 23,25,51,91–95. Such condition-independent components were not implemented in the models in Fig 1. Unlike in the models, pronounced choice-related activity occurred along more than one state-space direction (Fig. 3c, choice plane) and rotational dynamics within more than one plane. Moreover, rotational dynamics was observed also during the decision-epoch (Fig. 3c, jPC planes). As for the models in Fig. 1, it is not clear which of these features of the condition-averaged trajectories reflect the influence of inputs, recurrent dynamics, or both.
Residual dynamics in PFC
To better resolve the contributions of recurrent dynamics to the recorded responses, we characterized residual dynamics in PFC, by proceeding in two steps. First, we estimated a “dynamics subspace”, contained within the previously defined aligned subspace (Fig. 4a, Extended Data Figs. 2,5,7). The dynamics subspace was defined such that within it, but not outside of it, residuals at any given time are significantly correlated with residuals at previous or future times. Second, we exploited these correlations to estimate residual dynamics within the dynamics subspace, following the same approach as for responses simulated from the models above (Fig. 2e-g, Extended Data Fig. 2,8).
We found that residual dynamics in PFC was stable and decaying across the decision and movement epochs (Fig. 4b), as the largest estimated EV magnitudes were consistently smaller than 1 in both monkeys (Fig. 4e; p < 0.001, single tailed t-test, n= 144 data points across times, choices and configurations). The dynamics subspace was close to 8-dimensional in all configurations (Fig 4a, Extended Data Fig. 7,8) and was best aligned with directions that explained most task-related variance within the aligned subspace (Fig. 4a, largest dot products at small values along y-axis; Extended Data Fig. 5). Any directions lying outside the dynamics subspace can be thought of as being associated with an EV equal zero, meaning that perturbations along these directions completely decay within a single time step.
The EV magnitudes were strongly time-dependent. For all task configurations, the largest EV were attained during the decision epoch or the delay period preceding the saccade. These EV were associated with decay time-constants in the range 187-745ms during the decision period (0s to +0.8s following stimulus onset) and 110-913ms during the delay period (-0.5s to +0.3s relative to saccade onset) for monkey T (95% CI, medians = 352ms and 293ms; Fig. 4e, top), and 309-1064ms and 192-3586ms for monkey V (95% CI, medians = 489ms and 491ms; Fig. 4e, bottom). Concurrently with the saccade onset, the EV consistently underwent a strong contraction—the largest measured time constants at saccade onset fell to median values of 159ms in monkey T and 310ms in monkey V (Fig. 4e), implying that perturbations away from the average trajectory during movement quickly fall back to the trajectory.
These findings alone rule out several models of recurrent dynamics in PFC. Even the largest EV in PFC during the decision epoch are inconsistent with unstable dynamics (EV>1, Figs. 1a,2e; saddle point) and for the most part substantially smaller than what would be expected from persistent dynamics (EV≈1, Figs. 1a,2e; line attractor). Likewise, the small EV around the time of the saccade are inconsistent with purely rotational or funnel dynamics, which would both result in directions with very slow decay (EV≈1, Figs. 1b,2e; rotations and funnel). Rather, the inferred EV are consistent with quickly decaying recurrent dynamics (Figs. 1b,2e; point attractor).
The absence of strong rotational dynamics is bolstered by the finding that the largest estimated rotation frequencies are either close to zero or very small for most EV in both monkeys (Fig. 4f). We did observe a few EV with rotation frequency considerably larger than zero (≈0.5-1Hz) in monkey T (Fig. 4c). However, around the time of movement the associated EV magnitudes were small (e.g. time constants between 70-110ms, Fig. 4b; dark blue) implying that perturbations decay within 1/15th of a rotational cycle. Overall, these findings are inconsistent with the large rotation frequencies and slow decay expected for purely rotational recurrent dynamics (Fig. 2e,g; rotations).
Finally, the largest SV had a somewhat larger magnitude than the largest EV throughout both task epochs, particularly in monkey T (compare Fig 4e to 4g). This finding indicates that dynamics in PFC is non-normal, albeit only weakly. Even the largest SV are smaller than 1, implying that the non-normal recurrent dynamics does not amplify perturbations, it only transiently slows down their decay. The degree of non-normality, quantified as the discrepancy between the EV and the SV, followed a consistent time-course across animals and configurations, and was most pronounced around the time of the saccade (Fig. 4h).
Condition-averaged trajectories reflect time-dependent input contributions
Additional insights into the relative strengths of recurrent dynamics and inputs can be gained by comparing the properties of residual dynamics and condition-averaged trajectories. When inputs are weak, the trajectories mostly reflect the properties of the recurrent dynamics, which in turn results in distinct relations between trajectories and residual dynamics. For example, in the saddle-point and line-attractor models, the condition-averaged trajectories for the two choices diverge along a direction that is closely aligned with the eigenvector associated with the largest EV in the residual dynamics (Fig. 1a, left-most panels; horizontal direction; Fig. 2e). Similarly, in the funnel and rotation models, the condition-averaged trajectories rotate in the plane containing residual dynamics with EV close to 1 (Fig. 1b, left-most panels; Fig. 2e), or EV with large angular phase (Fig. 1b, left panels; Fig. 2g). When such relations are absent, two scenarios are possible. First, the neural trajectories may mostly be driven by a strong input (Fig. 1b,2e, point-attractor model: trajectories rotate, whereas residual dynamics is decaying and non-rotational). Second, the recurrent dynamics may implement strong non-normal amplification, where population trajectories can display pronounced excursions along directions that are largely orthogonal to the eigenvectors associated with the largest EV82,85,96,97. While the latter scenario is ruled out by the properties of the residual dynamics (Fig. 4g,h, SV≤1; no non-normal amplification), the former is not.
We quantified the relationship between residual dynamics and the condition-averaged trajectories in PFC (Fig. 5) as the subspace angle between the eigenvectors of the residual dynamics and the four activity subspaces that we defined based on condition-averaged trajectories (Fig. 3). To reveal relationships of the kind predicted by some of the decision and movement models, we sorted subspace angles, based either on the magnitude (Fig. 5a,b left half of x-axis) or rotation frequency of the associated EV (Fig. 5a,b, right half of x-axis). For each magnitude and angular phase, we also determined whether the measured subspace angles were significantly smaller (i.e. “aligned”) or larger (“mis-aligned”) than expected based on randomly chosen directions within the dynamics subspace (Fig. 5, crosses).
During the saccade epoch, subspace angles with the jPC planes showed no dependency on either the rotation frequency or the magnitude of the associated EV in both monkeys (Fig. 5a-d, bottom). In fact, the mean subspace angle obtained for any given EV magnitude or rotation frequency closely matched that expected from the null distribution (Fig. 5a-d, bottom; vertically aligned green and purple points on the left and right). The prominent rotations in the condition-averages during the saccade epoch (Fig. 3d) are thus not preferentially aligned with eigenvectors associated with EV of large magnitude or large rotation frequency. This finding is inconsistent with the predictions of the rotation and funnel models, and instead suggests a prominent role of inputs in driving saccade-related activity (Fig. 1b, point attractor).
On the other hand, the subspace angles were related to the properties of the residual dynamics during the decision-epoch, although the observed relations differed across animals. In monkey V, the choice plane is best aligned with the eigenvectors of the largest magnitude EV (Fig. 5c top, left half of x-axis; green points), consistent with a role of the recurrent dynamics in generating choice responses. This relation is less pronounced in monkey T (Fig. 5a top, left half of x-axis; green points), for which the residual dynamics was better aligned with the time-plane, capturing choice-independent variance, than with the choice plane (Fig. 5a top, left half of x-axis; purple points). Nonetheless, a pronounced relation between residual dynamics and condition-averaged responses was apparent in monkey T, although of an unexpected kind. The subspace angles with the time-plane (Fig. 5a top, right half of x-axis; purple points) and the jPC12 plane (Fig. 5b top, right half of x-axis; green points) showed a strong dependence on EV rotation frequency, suggesting that the rotational structure of the trajectories in those planes during the decision epoch reflects the influence of rotational recurrent dynamics.
In both monkeys, the properties of residual dynamics (Fig. 4) and its relation to condition-averaged trajectories (Fig. 5) thus suggest that recurrent dynamics substantially contributes to shaping the condition-averaged trajectories measured in PFC only during the decision-epoch (Fig. 3c). The large excursions in the trajectories observed during the movement epoch (Fig. 3d) instead seem more consistent with the influence of strong external inputs11.
Interpreting local residual dynamics in distributed cortical circuits
These above conclusions, however, are based on a comparison to simplified models of neural dynamics, for which inputs and recurrent contributions are well defined (Fig. 1). Biological circuits tend to be modular, i.e. are subdivided into areas, with both local recurrence within areas, as well as long-range, feedforward or feedback connections between areas98,99. At the level of any single area, a clear distinction between inputs and recurrent dynamics may then be challenging, raising the question of how residual dynamics should be interpreted when computations are distributed across many areas.
To address this question, we consider simulations of a two-area, non-linear, recurrent neural network previously proposed to explain the interplay of posterior parietal cortex (PPC) and PFC during decision-making and working-memory48. The network implements both local recurrence within each area (PPC and PFC), as well as long-range connectivity between the two areas. PPC is assumed to be upstream of PFC, as it alone receives an input encoding external stimuli. Here we consider only a limited set among all possible network configurations. First, the strength of local recurrence is set to be equal in both areas. Second, when feedback connections from PFC to PPC are present, their strength equals those of the feedforward connections from PPC to PFC.
Simulated responses of a random-dots task show choice-dependent and condition-independent components, both in PPC and PFC (Fig. 6a,d; choice and time modes). The EV of the residual dynamics, estimated locally in PPC or PFC, are typically time-dependent (Fig. 6b,e, Extended Data Fig. 9). In particular, the dynamics can change from stable (EV<1) to unstable (EV>1) after the input is turned on, reflecting the non-linear nature of these networks.
To assess the interaction of local recurrence and long-range connections, we focus on residuals dynamics estimated along the choice mode in each area (Fig. 6c,f, Extended Data Fig. 9). By design, the choice modes define the “communication subspace” between PPC and PFC in these networks20,48—the feedforward and feedback connections between areas are constructed such that activity along the choice mode in one area drives activity along the choice mode in the other area (Extended Data Fig. 9). We summarize the residual dynamics in each network with the peak magnitude of the EV along the choice mode achieved within a trial (Fig. 6c,f, Extended Data Fig. 9).
In networks lacking feedback between areas, the residual dynamics in PPC naturally only reflects the local recurrence, whereby the largest EV gradually increases with stronger local recurrence. (Fig. 6c, PPC). The residual dynamics in PFC closely resembles that in PPC (Fig. 6c, PFC), but this resemblance conceals a critical difference between the two areas. In PPC, the residual dynamics reflects the properties of the local recurrent dynamics. The same is not true in PFC, where any EV>1 mostly reflects recurrent dynamics implemented upstream, in PPC. Indeed, if the output of PPC is “shuffled” to remove any temporal correlations, while retaining its time-varying mean, the EV estimated in PFC fall below 1, indicating that recurrent dynamics in PFC is actually decaying in these networks (Fig. 6b, dashed). We refer to this effect as an “inflation” of the EV in PFC, due to the correlated input from PPC.
Such an inflation of local residual dynamics can occur whenever trial-by-trial variability in the inputs into an area displays correlations across time, as can be the case when the upstream areas themselves implement recurrent dynamics (Extended Data Figs. 10,11). This effect implies that the EV magnitudes we estimated in PFC (Fig. 4) set an upper limit to the “true” values one would observe based on local PFC recurrence alone. Notably, not just the magnitude of the estimated EV can be inflated, but also their rotation frequency (Extended Data Fig. 10b,d). Estimated EV with large rotation frequency could thus reflect rotational dynamics occurring locally, or that are implemented in areas upstream to the recorded area (Extended Data Fig. 10d,e).
In networks with long-range feedback, the residual dynamics in PPC and in PFC reflects both the strength of local recurrence and of long-range connections, whereby reduced local recurrence can be entirely compensated by increased global feedback (Fig. 6f). Unlike in the feedforward networks, where the choice results entirely from dynamics unfolding locally in PPC, here the choice dynamics reflects a process distributed across both areas.
Overall, these simulations show that local residual dynamics in an area cannot be assumed to only reflect local recurrence in that area, as very different combinations of local and long-range connectivity can result in virtually indistinguishable residual dynamics at the level of single areas (Fig. 6a,d vs. b,e). At the same time, these analyses also demonstrate that local residual dynamics can resolve recurrent computations implemented outside of the recorded area, as long as they are unfolding within the output subspace of an upstream area.
Global residual dynamics resolves local and global recurrent computations
The simulations in Fig. 6 imply that the properties of recurrent dynamics in PFC can be constrained, but are not unambiguously revealed, by local estimates of residual dynamics (see Discussion). However, we find that detailed insights into the interaction of local and long-range recurrence are possible when considering the global residual dynamics, which is estimated from recordings across all areas in a network.
We estimated global residual dynamics from the concurrent, pooled responses simulated in PPC and PFC, for the two example networks with long-range feedforward and feedback connections (Fig. 7). EV magnitudes are qualitatively similar in the two networks, with one EV unstable (EV>1), one persistent (EV≈1), and the others decaying (EV<1; Fig. 7a). The number of global EV does not robustly distinguish between networks, as it reflects a somewhat arbitrary cutoff in the dimensions to include in the dynamics subspace (excluded dimensions effectively have EV=0).
Critically, the alignment between the eigenvectors of the global EV and the local task-activity subspaces can distinguish between the two networks. As above (Fig. 5), we quantified the alignment as the angle between the estimated global eigenvectors and the local choice and time modes in PPC and PFC (Fig. 7b; gray: feedforward, black: feedback). Eigenvectors can be either “shared” across areas, or “private” to an area, depending on whether they have substantial projections (i.e. angle<90) onto modes in both areas or only a single area. For example, EV1 is shared in both networks, albeit to different degrees, whereas EV2 is consistently private to PPC. While several global eigenvectors are aligned differently with the PPC and PFC modes in the two networks (Fig. 7b, EV1 and EV3), such differences are not evident at the level of local residual dynamics (Fig. 7c).
Global residual dynamics can distinguish between the two networks because variability evolves differently within and across areas depending on the connectivity between areas. To explore the possible nature of these differences, we first consider the effect of perturbations in two simple models implementing time-independent, linear dynamics (Fig. 7d), which mimics key properties of the inferred global dynamics (Fig. 7b). We considered activity that is only two-dimensional, whereby the two cardinal dimensions represent the choice modes in PPC and PFC, respectively (Fig. 7d). The two models differ in the arrangement of the two eigenvectors of the dynamics, but not in the magnitudes of the associated EV. In the “feedforward” model, an unstable eigenvector projects mostly onto the PPC choice mode, while a stable eigenvector is aligned with the PFC choice mode (Fig. 7d, top; EV1 and EV3; similar to the corresponding gray points in Fig. 7b). In the feedback model, both the unstable and stable eigenvectors have large projections onto the PPC and PFC choice modes (Fig. 7d, bottom; EV1 and EV4; similar to the corresponding black points in Fig. 7b).
We mimicked a local perturbation either in PPC or PFC by initializing activity along the corresponding choice mode (Fig. 7d, left; black points), and then letting activity evolve based on the linear dynamics determined by the respective EV (Fig 7d, left; white points).
These simple models exemplify how the arrangement of global eigenvectors determines the directionality of the communication between areas. In the feedforward model, a PPC perturbation causes expanding activity in PPC that propagates to PFC, whereas a PFC perturbation decays in PFC, and does not propagate to PPC (Fig. 7d, top, right column). This unidirectional communication results from non-normal dynamics, as EV1 is shared, while EV3 is private to PFC (Fig. 7d, top; EV1 not orthogonal to EV3). In the feedback model, perturbations in either PPC and PFC propagate to the other area (Fig. 7d, bottom, right column). Such bidirectional communication results from normal dynamics, and the fact that both EV1 and EV4 are shared equally between PPC and PFC.
Notably, the existence of bidirectional communication is also reflected in the activity of the perturbed area. Somewhat counter-intuitively, activity in the area that was perturbed initially decays, and expands only later; activity in the unperturbed area does not show this dip (Fig. 7d, feedback; PPC and PFC activity in right panels). This dip in activity occurs because any local perturbation is only partially aligned with the shared, unstable direction (EV1). Initially, activity in the perturbed area then mostly reflects the rapidly decaying component of activity along the second, global eigenvector (EV4).
Inferring global dynamics with local causal perturbations
We directly verified the insights from these simple linear models by simulating the effect of causal perturbations in the example two-area networks (Fig. 8). We applied local perturbations, either in PPC or PFC, by “injecting” an activity pattern corresponding either to the choice mode or the time mode in each area. For each trial, we applied a brief perturbation at one of six different times after stimulus onset, and then let the activity evolve under the influence of the recurrent dynamics and the input. We visualize the effect of a given perturbation as the time-varying norm of the population activity in PPC and PFC for a brief time-window following the onset of the perturbation, averaged over many trials (Fig. 8b-c,e-f; a group of three connected points; analogous to Fig. 7d). The effects of a perturbation depend on the time at which it was applied (Fig. 8b-c,e-f, compare time-courses within each panel), reflecting the time-varying dynamics in these networks (Fig. 7a).
For perturbations applied late in the trial, when dynamics is unstable (Fig. 7a, EV>1), perturbations of the choice modes result in activity that largely matches the dynamics of the simple models above (Fig. 7d). In the feedforward network, PPC perturbations lead to expanding activity in PPC and PFC (Fig. 8b,c; top-left, green), whereas PFC perturbations lead to decaying activity in PFC (Fig. 8c, bottom-left) and no activity in PPC (Fig. 8b, bottom-left). In the feedback network, PPC and PFC perturbations lead to a dip in activity in the perturbed area (Fig. 8e, top-left and Fig. 8f, bottom-left) and to expanding activity in the non-perturbed area (Fig. 8f, top-left and Fig. 8e, bottom-left), as in the corresponding simple model (Fig. 7d, feedback). All these effects are specific to perturbation along the choice modes— perturbations along the time-mode, in either area, result in very different, consistently decaying dynamics (Fig. 8b-c,e-f; purple color).
These varied effects of causal perturbations can be predicted quite accurately based entirely on our estimates of the global residual dynamics (Fig. 7a-b). The predicted time-course of activity following a perturbation at least qualitatively matches the simulated one, for all types of perturbations (Fig. 8b-c,e-f, black). Predictions based on local estimates of residual dynamics fare much worse overall, but the failures are nonetheless informative about the underlying network (Fig. 8b-c,e-f, gray). For example, the inflation of local PFC residual dynamics in the feedforward network (Fig. 6b) leads to the erroneous prediction that PFC perturbations result in expanding, rather than decaying, PFC activity (Fig. 8c, bottom-left, gray). In the feedback model, predictions based on local residual dynamics instead fail to account for the dip in activity in the perturbed area (Fig. 8e, top-left, Fig. 8f, bottom-left) and underestimate the increase in activity in the unperturbed area (gray; Fig. 8f, top-left, Fig. 8e, bottom-left). Both failures reflect the existence of a global, shared unstable direction, which local residual dynamics cannot adequately capture.
Discussion
It has long been recognized that trial-by-trial variability in neural activity can provide insights into population-level computations12–22. Residual dynamics amounts to a complete, quantitative description of the dynamics of trial-by-trial variability at the level of a neural population. Residual dynamics tightly relates to the recurrent computations implemented in the underlying neural circuits, and is capable of resolving fine differences in dynamics across state-space locations and time. This fine resolution allows one to describe dynamics that are globally non-linear100,101, through a series of local approximations. Unlike past statistical approaches that directly model single-trial dynamics68,69,71, residual dynamics completely discounts the component of neural responses that is repeatable across trials of a given task condition. As a result, residual dynamics can be estimated with more easily interpretable models than the dynamics of the full, single-trial neural responses.
The properties of global residual dynamics, based on recordings distributed across a network of inter-connected areas, can potentially resolve contributions of local, within-area recurrence and long-range, between-area connections (Fig. 7). The resulting description of dynamics in terms of modes (i.e., eigenvectors) that are shared across areas98, or private to a single area, appears plausible based on the past identification of communication- and null-subspaces between areas20,102,103—an eigenvector that is shared between two areas lies within their communication subspace, whereas one that is private lies outside of it, and potentially within the null-space of either area. Global residual dynamics, however, goes beyond a static description based on such subspaces, as it can capture also the dynamics of the responses (Fig. 8) resulting from unidirectional or bidirectional communication between areas (Fig. 7d, top vs. bottom). Local residual dynamics in a single area, of the kind we describe for PFC, instead cannot readily distinguish between local and global contributions to observed neural responses. Any recurrent dynamics unfolding within the communication subspace of two areas will be reflected in the local dynamics of both areas, irrespective of the directionality of the communication between the areas (feedforward or feedback, Fig. 6c,f).
Nonetheless, even our local estimates of PFC residual dynamics provide constraints on the properties of recurrent dynamics in PFC, and on the nature of the computations underlying decision-making and movement generation. For one, the largest estimated time constants provide an upper bound on the time-constants of the local recurrent dynamics in PFC (Fig. 4e; 324ms and 510ms in monkeys T and V; medians), as any upstream contribution to PFC responses would have inflated these estimates (Fig. 6b; Extended Data Fig. 10,11). Recurrent dynamics in PFC is thus slow98,104, but stable throughout the decision and movement epochs.
This finding does not rule out that the decision-process leading to the monkeys’ choices involves unstable or line-attractor dynamics (Fig. 1a), but those dynamics would have to unfold in areas upstream of PFC80,105, and at least partly outside their communication subspace with PFC. The estimated time-constants would reflect the dynamics of the decision-process if that process unfolded either in PFC alone, or within its communication subspace with other areas (as for all networks in Fig. 6). In such scenarios, our estimates would imply a leaky decision-process, whereby late evidence affects choice more strongly than early evidence. In practice though, monkeys are thought to terminate the accumulation of evidence early in the trial, when a decision-threshold is reached106, which would reduce the behavioral effects of any leaks in the accumulation. Notably, a recent study hypothesized that the termination of evidence accumulation coincides with the onset of rotational dynamics in PFC 107. In our study, condition-independent, rotational dynamics during the decision-epoch also stands out, as in monkey T it is the component of the recorded activity that can be best explained as resulting from recurrent computations (Fig. 5). Irrespective of the possible contributions of PFC to the process underlying the monkeys’ choices, this finding may be indicative of a broader role for PFC in governing transitions between cognitive states107,108, e.g. the transition from an uncommitted to a committed state.
Around the time of the saccade, PFC residual dynamics is quickly decaying, largely non-rotational, and only weakly non-normal, implying that PFC does not implement rotational45,46, funnel47,61, or strongly non-normal82,85 recurrent dynamics of the kind previously proposed to explain movement activity in cortex. Rotational and funnel dynamics are also unlikely to be implemented in an upstream area driving PFC movement responses through a communication subspace, since the signatures of those dynamics would then also appear in PFC residuals (Fig 6, Extended Data Fig. 10). Strong non-normal dynamics in an upstream area, however, could explain the residual dynamics and condition-averages observed in PFC. Non-normal systems can generate large activity transients along directions with only a small projection onto the activity subspace containing the slowest dynamics97. If the output from such an upstream area was partially aligned with the activity transients, but orthogonal to the slow dynamics, it could drive strong “input-driven” movement-related activity in PFC without revealing the signatures of the strongly non-normal dynamics that created it. Notably, the apparent absence of rotations in PFC recurrent dynamics during saccades does not rule out that such dynamics occurs in premotor and motor areas involved in hand-reaches43,45. The neural mechanisms underlying saccades and reaches may well be distinct, considering the substantial differences in the anatomy of the involved structures88,89.
A complementary approach to distinguishing between the above interpretations of PFC function, beyond characterizing global residual dynamics, would involve combining local estimates of residual dynamics with targeted causal perturbations11,23–30. Residual dynamics naturally leads to predictions of the consequences of such perturbations, and failures of the predictions can be diagnostic of the underlying long-range connectivity (Fig. 8). Most useful in this respect are small perturbations that probe the intrinsic manifold explored by the neural variability27,31,32.
Residual dynamics and the structure of variability may also speak to specific biological constraints at play in neural circuits. The observation of eigenvalues that are smaller, but close to 1 during the decision-epoch is consistent with the underlying neural circuit operating near a critical regime, resulting in large variability and sensitivity to inputs109–112. Variability at the level of single neurons is transiently reduced at the time of stimulus and movement onset (Extended Data Fig. 12), potentially reflecting the widespread quenching of variability across cortex in response to task events13,113,114. Near-critical dynamics, non-normality, and variability quenching are thought to emerge naturally in balanced excitation-inhibition (E-I) networks115–117. A disruption of E-I balance by the onset of an input could potentially lead to contracting dynamics, and thus reduced variability. Notably, the observed reduction in variability in PFC coincides with contracting dynamics at movement onset, but not at stimulus onset (Extended Data Fig. 12), suggesting that such E-I networks may have to be adapted to fully capture the interactions of internal dynamics, inputs, and variability we observed in PFC.
Author Contributions
A.R.G and V.M conceived and designed the study. A.R.G developed the methods and performed the analyses, with input from M.S. and V.M. A.R.G and V.M wrote the manuscript. All authors were involved in discussing the results and the manuscript.
Funding
This work was funded by Swiss National Science Foundation (Award PP00P3-157539, VM), the Simons Foundation (SCGB 328189 and 543013, VM; SCGB 543039, MS), the Swiss Primate Competence Center in Research (VM), the Gatsby Charitable Foundation (MS), the Howard Hughes Medical Institute (William Newsome), and the Air Force Research Laboratory (William Newsome).
Extended Data Figures
Acknowledgements
We thank John Reppas and William Newsome for the data collection. We thank Kevan Martin and all members of the Mante Lab for their valuable feedback, as well as Nicolas Meirhaeghe, Lea Duncker and Mehrdad Jazayeri for discussions and comments on the manuscript.
References
- 1.↵
- 2.
- 3.
- 4.↵
- 5.↵
- 6.
- 7.↵
- 8.↵
- 9.
- 10.↵
- 11.↵
- 12.↵
- 13.↵
- 14.
- 15.
- 16.
- 17.↵
- 18.
- 19.
- 20.↵
- 21.
- 22.↵
- 23.↵
- 24.
- 25.↵
- 26.
- 27.↵
- 28.↵
- 29.
- 30.↵
- 31.↵
- 32.↵
- 33.↵
- 34.↵
- 35.↵
- 36.↵
- 37.
- 38.↵
- 39.↵
- 40.↵
- 41.↵
- 42.↵
- 43.↵
- 44.
- 45.↵
- 46.↵
- 47.↵
- 48.↵
- 49.↵
- 50.
- 51.↵
- 52.
- 53.
- 54.
- 55.↵
- 56.↵
- 57.
- 58.
- 59.
- 60.↵
- 61.↵
- 62.
- 63.
- 64.
- 65.
- 66.↵
- 67.↵
- 68.↵
- 69.↵
- 70.
- 71.↵
- 72.
- 73.
- 74.
- 75.↵
- 76.↵
- 77.↵
- 78.
- 79.
- 80.↵
- 81.↵
- 82.↵
- 83.
- 84.
- 85.↵
- 86.
- 87.↵
- 88.↵
- 89.↵
- 90.↵
- 91.↵
- 92.
- 93.
- 94.
- 95.↵
- 96.↵
- 97.↵
- 98.↵
- 99.↵
- 100.↵
- 101.↵
- 102.↵
- 103.↵
- 104.↵
- 105.↵
- 106.↵
- 107.↵
- 108.↵
- 109.↵
- 110.
- 111.
- 112.↵
- 113.↵
- 114.↵
- 115.↵
- 116.
- 117.↵