## Abstract

The complex activity of neural populations in the Prefrontal Cortex (PFC) is a hallmark of high-order cognitive processes. How these rich cortical dynamics emerge and give rise to neural computations is largely unknown. Here, we infer models of neural population dynamics that explain how PFC circuits of monkeys may select and integrate relevant sensory inputs during context-dependent perceptual decisions. A class of models implementing linear dynamics accurately captured the rich features of the recorded PFC responses. These models fitted the neural activity nearly as well as a factorization of population responses that had the flexibility to capture non-linear temporal patterns, suggesting that linear dynamics is sufficient to recapitulate the complex PFC responses in each context. Two distinct mechanisms of input selection and integration were consistent with the PFC data. One mechanism implemented recurrent dynamics that differed between contexts, the other a subtle modulation of the inputs across contexts. The two mechanisms made different predictions about the contribution of non-normal recurrent dynamics in transiently amplifying and selectively integrating the inputs. In both mechanisms the inputs were inferred directly from the data and spanned multi-dimensional input subspaces. Input integration likewise consistently involved high-dimensional dynamics that unfolded in two distinct phases, corresponding to integration on fast and slow time-scales. Our study offers a principled framework to link the activity of neural populations to computation and to find mechanistic descriptions of neural processes that are consistent with the rich dynamics implemented by neural circuits.

## Introduction

A fascinating aspect of our daily existence is that, in a blink of an eye, we can effortlessly change our course of action, switch between tasks or wander in between lines of thought. To achieve this flexibility, brain circuits must be endowed with mechanisms to perform context-dependent computations, so that behavior is quickly adapted to each situation and the correct decisions can be taken. The mechanisms underlying this flexibility are still poorly understood.

A brain structure known to mediate flexible computations is the Prefrontal Cortex (PFC)^{1}. PFC is part of an extensive and highly distributed network of cortical and subcortical areas comprising the decision-making circuitry of the brain^{2}. It is involved in the generation of complex behaviors such as planning, selective attention and executive control^{3,4}. PFC is thought to hold the representation of goals, contexts and task rules^{5,6} and in primates is required to switch behaviors according to different task instructions^{7}. Finally, PFC’s crucial role in ignoring task distractors suggests that it actively filters out irrelevant information^{8,9}. This makes PFC of special importance for studying contextual decision-making.

Previous work suggested that flexible prefrontal computations emerge from the concerted interaction of large, interacting neural populations^{1}. Surprisingly, during contextual decisions requiring monkeys to integrate noisy sensory information towards a choice, irrelevant information did not appear to be gated at the level of inputs into PFC. Instead, irrelevant inputs may be dynamically discarded through recurrent computations occurring within PFC. A possible mechanism for such dynamical gating was revealed by reverse-engineering recurrent neural networks (RNNs) trained to solve the same contextual decision-making task as the monkeys. Remarkably, the trained RNNs reproduced key features of the PFC population activity, even though the networks were not explicitly designed to match the dynamics of the data. The match with the recorded data, however, was only qualitative, as the networks failed to reproduce many aspects of the rich and heterogeneous responses of individual PFC neurons. This raises the question of whether a model that captured the complex PFC dynamics would rely on the same contextual decision-making mechanism as the RNNs.

In this study, we took the approach of fitting linear dynamical system models (LDS) directly to the PFC data, allowing us to infer interpretable linear systems that approximate the neural population activity in each context. We characterized the nature of computations implemented in each context by analysing the properties of such models, whose dynamics closely matched those of the PFC population. To validate our assumption of linear dynamics, we compared the LDS to a novel low-rank factorization of the data, Tensor Factor Regression (TFR), which can capture non-linear dynamics. Both models performed comparably, implying that a linear model is sufficient to explain the PFC activity in a given context.

We fitted different LDS model classes corresponding to different hypotheses about the nature of context-dependent computations in PFC. One class could implement context-dependent recurrent dynamics but received fixed inputs, mimicking the design of RNNs developed in past work^{1}. Another class had fixed recurrent dynamics, but could implement context-dependent inputs. In both models, we inferred external input signals directly from the data. Surprisingly, these two model classes explained the PFC responses similarly well, meaning that both contextual decision-making mechanisms are consistent with the data. Both mechanisms shared some features with the RNN solution, but also differed from it in important ways, thus revealing previously unknown properties of contextual decision-making computations in PFC.

Our data-driven approach to analyzing neural dynamics, based on fitting LDS models to neural population responses, can be applied across different brain areas, neural data sets, and computational mechanisms, providing a general tool to test specific hypothesis about the nature of computations implemented by neural circuits.

## Results

We analysed PFC recordings from two monkeys performing a contextual version of the classic random dots motion task^{1,10}. The monkeys had to report the overall color or motion of the dots, depending on context (Fig. 1a). Since both types of sensory evidence were simultaneously presented, the monkeys had to actively ignore the irrelevant sensory input in order to form a decision based only on the relevant input. We analysed only correct trials and focused on the random dots presentation period, during which the motion and color evidence needed for a correct decision were presented^{1}. In the next sections, we present an in-depth analysis of the PFC data from one of the monkeys (monkey A). Findings from monkey F are presented in the supplementary material, and confirm the key insights gained from monkey A (Supplementary Figs. 6 to 10).

### PFC dynamics in each context are well approximated by a linear system

In order to infer the mechanisms underlying PFC population dynamics, we fitted several LDS models to the PFC responses (Fig. 1b). Each LDS was parameterised by three distinct components: a dynamics matrix *A*, which determined the recurrent contribution to the evolution of the low-dimensional (low-d) latent activity state **x**(*t*); by external motion and color inputs **u**_{m}(*t*) and **u**_{m}(*t*); and by motion and color input subspaces *B _{m}* and

*B*, which specified the dimensions along which the external inputs modulated the state dynamics. The dynamics matrix and the input subspaces were fixed over time, whereas the external inputs could be time-varying. An orthonormal “loading” matrix

_{m}*C*mapped the low-d latent activity

**x**(

*t*) to the high-dimensional (high-d) “observations”

**ŷ**

_{PFC}(

*t*), i.e. the condition-averaged z-scored peri-stimulus-time-histograms (PSTHs) of individual units in PFC. Therefore, the observations were reconstructed based on a linear combination of low-d latent activity, which approximated the dynamics of the high-d neural population.

We captured the observed changes in activity across contexts^{1} by fitting an LDS model jointly to the PFC data from the two contexts. Some of the model parameters varied across contexts, while the remainder were shared across contexts. We implemented several distinct classes of LDS, which differed with respect to their context-dependent parameters. In a first class, the dynamics matrix *A ^{cx}* could differ across contexts (Fig. 1b,

*cx*= mot/col context), whereas the input parameters were fixed. In a second class, the dynamics matrix was fixed, but the motion and color subspaces were allowed to vary across contexts. Both model classes can process inputs flexibly, but do so based on different mechanisms.

The *A ^{cx}*,

*B*model class retains some of the key properties of previously proposed RNNs

^{1}. As in the RNNs, the motion and color inputs are fixed across contexts, meaning that any context-dependent computations must be achieved by the recurrent dynamics (Fig. 1c). The

*A, B*model class instead relies on contextually modulated inputs, a mechanism that appeared unlikely based on past analyses of the PFC responses

^{cx}^{1}. Both model classes differ from the RNNs in several ways. First, in the LDS all parameters were learned from the data (Fig. 1b, grey boxes), including the external, time-varying inputs

**u**

_{m}(

*t*) and

**u**

_{m}(

*t*). The RNN instead was trained on the task, with hand-crafted external inputs that were constant over time. Second, the LDS could learn multi-dimensional input subspaces

*B*. Such subspaces could capture rich activity patterns arising under direct influence of the external inputs, which may be required to explain some aspects of the data

_{m,c}^{11,12}. In the RNNs, the inputs instead were one-dimensional. Importantly, we fitted the LDS models with a regularization favoring weak inputs, to avoid solutions that relied entirely on input driven activity. Activity patterns that do not directly represent the motion and color coherence, such as the integrated relevant evidence or activity related to the passage of time, would then have to emerge through the transformation of the inputs by the recurrent dynamics in all LDS models.

Surprisingly, we found that the two LDS model classes could explain the PFC responses similarly well (Fig. 2a, *A ^{cx}, B* and

*A, B*; cold color lines), implying that two very different mechanisms could explain the observed activity. A third model class that had contextual flexibility in both the recurrent dynamics and the inputs (referred to as

^{cx}*A*) did not improve the fits. A model that could change only the initial conditions across contexts (Methods), but not the recurrent dynamics or the inputs (referred to as

^{cx}, B^{cx}*A, B*) instead performed significantly worse. We estimated the dimensionality of the latent dynamics and inputs based on generalization performance using leave-one-condition-out cross-validation (LOOCV

^{13}, Fig. 2b). All models performed substantially better for input dimensionality higher than 1D (Fig. 2a), meaning that they required multi-dimensional input signals. The best performing LDS models required three dimensions for both the color and motion inputs. The LDS models needed between 13 and 18 latent dimensions to best fit the data (Supplementary Table 1), many more than the 4 dimensions required to describe the task (motion, color, context, and decision).

Several lines of evidence show that the two best LDS model classes provide accurate descriptions of the PFC responses. First, the two LDS models closely approximated the highly heterogeneous responses of individual PFC neurons (Extended Data Fig. 1). The fits captured a substantial fraction of the data variance (27%, corresponding to MSE=0.73 on z-scored responses, Fig. 2b) even though we did not smooth nor “de-noise” the data^{1}. We included all neurons in the fits, even those with weak, sparse responses that could only be poorly captured by the models (Extended Data Fig. 1a,b, firing rates < 1Hz). Furthermore, we did not optimize our models to match the noise statistics of the data, since the fitted responses were trial-averaged.

Second, the best LDS model classes performed comparably to a more powerful model class that we refer to as Tensor Factor Regression (TFR). TFR is based on a new low-rank factorization of the data that partitions the data tensor into several low-d tensors, including a core tensor and an input tensor (Fig. 2c). The core tensor can be learned with independent parameters at each point in time, which provides TFR with a greater flexibility to capture temporal patterns compared to the LDS models, which are constrained to generate linear dynamics. The LDS classes are nested within the TFR model class (Methods), which places the two types of models on an equal footing and avoids potential pitfalls that can arise when performing model comparison across different model classes^{14}.

The observation that the LDS and TFR models performed comparably implies that the additional flexibility of the TFR model was not necessary to explain the data. PFC activity in each context thus appears to be well approximated by linear dynamical system models. The TFR model incorporated input parameters and contextual constraints equivalent to those of the LDS (Fig. 2c, Methods) and achieved comparable performance for a wide range of input and latent dimensionalities (Fig. 2a,b, warm color lines). The TFR model required a similar range of latent dimensionality than the LDS (13-18, Supplementary Table 2). For the TFR model, however, the optimal inputs were 2-dimensional, compared to the 3-dimensional inputs required by the LDS. This difference could imply that the LDS needs an extra input dimension to overcome the limitations of the linear dynamical constraints, a possibility that should be taken into account when interpreting the parameters of the LDS model.

Third, the best LDS models qualitatively captured salient features of the population dynamics equally well. In particular, both the *A, B ^{cx}* and the

*A*models reproduced the rich PFC dynamics revealed by population trajectories in the low-d activity subspace capturing most variance due to motion, color and choice

^{cx}, B^{1}(Fig. 3). The TFR model resulted in comparable fits, both at the population (Extended Data Fig. 2a) and single neuron level (Extended Data Fig. 1).

The good qualitative match between the low-d PFC trajectories and the fits of the *A, B ^{cx}* model are somewhat surprising. By design, the task-related activity subspace capturing most variance due to motion, color, and choice can distinguish between computations that rely on inputs that are stable across contexts from computations implementing a strong suppression of the irrelevant input

^{1}. Computations relying on stable inputs result in characteristic features in the trajectories, in particular how they depend on the strength of the motion and color inputs (Fig. 3a). These features approximately occur in the measured PFC trajectories (Fig. 3b), and indeed are reproduced by the

*A*model (Fig. 3d). However, the same features are also reproduced by the

^{cx}, B*A, B*model (Fig. 3c), which by construction must rely on inputs that are variable across contexts.

^{cx}One conclusion from these analyses is that the properties of population trajectories in the considered low-d activity subspace are not sufficient to rule out computations that rely on variable inputs across contexts. The very close similarity between trajectories for the two LDS models suggests that the strength of input modulation required to explain the PFC responses is likely small. To understand how such small input modulations may support context-dependent integration in PFC, below we first separately characterize the inputs and recurrent dynamics in the *A, B ^{cx}* and the

*A*models, and then ask how their combined effects can account for contextual integration in PFC.

^{cx}, B### Input signals span curved manifolds and are largely stable across contexts

To better understand the mechanism of contextual integration implemented by the two LDS model classes, we fitted 100 models for each class, with random initialization and with latent and input dimensionality set by cross-validation (Fig. 2a, 3D inputs and latent dims 18 and 16). As we found similar model parameters across random initializations, we report parameter averages across the 100 models.

The context-dependent nature of computations in these models can be easily appreciated by considering the time-dependent norm of the latent activity, and . For each context, we computed this norm for activity in response to only one of the inferred inputs, with the other input set to zero (Fig. 4a). Here we refer to the resulting activity as the “output” of the LDS. In both models, the norm of the output activity increases over time within the trial and is much larger for the relevant compared to the irrelevant input, reflecting context-dependent integration (Fig. 4b,c, bottom panels, thick dotted vs. thin lines; significance, Wilcoxon rank-sum test, p<0.001 across 100 models, green bars; with 91 ± 2% increase for mot, 95 ± 3% for col in the *A, B ^{cx}* model, and 94 ± 3% increase for mot, 93 ± 3% for col in the

*A*model, mean ± std across 100 models). The predicted norms are essentially identical across the two models, in agreement with the similarity of the low-d trajectories they produce (Fig. 3c,d). In the

^{cx}, B*A*model this context dependence is entirely due to differences in the recurrent dynamics across contexts. In the

^{cx}, B*A, B*model, the difference must entirely reflect contextual modulation of the overall input strength, of the input direction, or a combination of the two.

^{cx}We first considered the time and context-dependence of the input strength, which we defined as and (motion and color strengths, respectively). Input strength is subtly different across the two models (Fig. 4b,c, top panels). After their initial rise at stimulus onset, the inputs were approximately sustained in the *A, B ^{cx}* model, but somewhat transient in the

*A*model. The strength of the motion and color inputs was similar, but both were overall weaker in the

^{cx}, B*A*compared to the

^{cx}, B*A, B*model. The finding that the inputs differ across models, whereas the output norm does not, implies that the recurrent dynamics must also operate differently in the two models. In the

^{cx}*A*model, input strength was the same across contexts, by definition. In the

^{cx}, B*A, B*model, input strength was different across contexts (Fig. 4b, top panels, thick dotted vs. thin lines, Wilcoxon rank-sum test, p<0.001, green bars), but only modestly, with the irrelevant inputs being slightly weaker than the relevant ones (38 ± 14% decrease for mot, 22 ± 8% for col, averaged across all time points starting at t=200ms, mean ± std across 100 models).

^{cx}Next, we considered the input direction, by characterizing the structure of coherence representations within the inferred input subspaces. Even though our cross-validation procedure consistently inferred 3-dimensional input subspaces, we found that most of the inferred input variance was contained in a 2D plane (Extended Data Fig. 3a). This plane was spanned by dimensions that captured, respectively, variance related to input coherence (mot, col) and coherence magnitude (|mot|, |col|). As a result, input coherence was represented along a curved 1D-manifold within the input plane^{15}. Similar curved representations were found by the two models (Fig. 4d,e) and for both contexts in the *A, B ^{cx}* model (Fig. 4d). Analogous 2D coherence representations were present also in the PFC data (Extended Data Fig. 3b), whereas the 3rd input dimension contained very little input and data variance (Extended Data Fig. 4c,f). In fact, the LDS models with 2D inputs were very close in performance to the 3D models (Fig. 2a) and captured the population trajectories nearly as well (Extended Data Fig. 2c,d). 1D input models, on the contrary, performed worse (Fig. 2a) and did not capture the trajectories along the input dimensions as well (Extended Data Fig. 2e,f).

The 2D input planes identified across the two model classes were highly aligned (16-31°, for averaged dimensions across 100 models from each model class, Extended Data Fig. 3c), an effect not expected by chance (Extended Data Fig. 3d). In the *A, B ^{cx}* model, the motion and color planes varied across contexts, but only modestly (33° ± 10 mot, 46° ± 16 |mot|, 25° ± 5 col, 27° ± 9 |col| dims, mean ± std across 100 models, Fig. 4d cartoons, Extended Data Fig. 3c) and less than expected by chance (Extended Data Fig. 3d). These relatively small changes in input direction across contexts, together with the concurrent, modest change in input strength (Fig. 4b, top), are entirely responsible for the context-dependent changes in the output of the

*A, B*model (Fig. 4b, bottom).

^{cx}In both models, the time-course (Fig. 4b,c, top) and structure of the inputs (Fig. 4d,e) is thus relatively simple, with most of the variance captured by a 2D-input subspace. This finding alleviates a possible confound inherent in fitting an LDS with time-dependent inputs. In principle, the fitted inputs could be very rich, and effectively approximate on their own the dynamics of a very complex, non-linear dynamical system. Considering that we retrieved inputs that are of much lower dimensionality than the recurrent dynamics (3D vs. 16-18D), such a scenario appears unlikely. Indeed, we find that the same classes of LDS models can be refit to the data with only a small drop in performance when their inputs are constrained to be fixed over time (Extended Data Fig. 5a, Supplementary Note 1). The observed complexity of PFC responses thus need not be inherited from the external inputs, but rather can be explained as resulting from approximately linear, time-dependent recurrent dynamics.

### Input integration relies on high-dimensional linear dynamics

To reveal the contributions of the recurrent dynamics to selective integration, we follow an approach motivated by the original analysis of RNNs trained to solve the same task as the monkeys^{1}. The computation implemented by the RNNs can be fully understood at the level of local linear approximations of the dynamics (Fig. 5a, left panel). Specifically, selective integration reflects four key features of the linear approximations. First, the largest eigenvalue of the dynamics is close to one, with the rest of the eigenvalues being much smaller than one, implying that input integration is implemented as movement along a line-attractor^{16}. Second, inputs are selected for integration based on their alignment with the leading left-eigenvector of the dynamics (the “input-mode” associated with the largest eigenvalue, i.e. slowest dynamics). The direction of this left-eigenvector is context-dependent, such that it is orthogonal to the contextually irrelevant input direction. Third, the direction of the leading right-eigenvector of the dynamics (the “output-mode” associated with the largest eigenvalue), which determines the direction of the line-attractor, is fixed across contexts. Fourth, the dynamics is non-normal, as the leading right and left eigenvectors have different directions. We collectively refer to these features of the linear approximations as the “RNN-mechanism” and below compare them to the dynamics of the fitted models.

We computed the eigenspectrum of the dynamics matrices, and in both models found multiple slow dimensions associated with time-constants that were relatively long compared to the duration of a trial (eigenvalues with norm |*λ*| > 0.8, or decay time-constant *τ* > 224ms, while the trial lasts 750 ms, Methods, Fig. 5b). The *A, B ^{cx}* model had a larger fraction of slow modes than the

*A*model (55 ± 7% vs. 35 ± 8% mot cx/ 41 ± 8% col cx, mean±std across 100 models). Many of the eigenvalues were imaginary, implying that the inferred recurrent dynamics was rotational

^{cx}, B^{17}(Extended Data Fig. 6a,b). The largest eigenvalue was 0.98 ± 0.02 for the

*A, B*model and 0.96 ± 0.03 / 0.99 ± 0.03 (mot/col cx) for the

^{cx}*A*model (mean norm±std across 100 models), corresponding to average decay time-constants of 2.5 and 1.2 / 5 seconds. While not strictly compatible with a line-attractor (an eigenvalue of 1), these time-constants are much longer than the duration of the trial. Additionally, the LDS models implemented many other slow modes, some with time-constants comparable to the duration of the trial (Fig. 5b). The large number of slow modes provides a first indication that PFC dynamics may be higher-dimensional than would be predicted by the RNN-mechanism.

^{cx}, BWe assessed context-dependent relations between the recurrent dynamics and the inputs by focusing on the coherence component of each input, while ignoring the absolute-coherence component (Fig. 4d,e). In the considered linear models, only the coherence component of the input can contribute to choice-dependent responses. We first examined the “load” (the non-normalized projection, Methods) of the coherence input onto each left-eigenvector, computed separately at each time instant and then averaged over the entire trial (Fig. 5c). Consistent with the RNN mechanism, the input load onto the left-eigenvectors was overall larger for the relevant vs. the irrelevant input, in both types of models (Fig. 5c, green bars, Wilcoxon rank-sum test, p<0.05). Furthermore, the load was close to zero for the irrelevant input along the slowest mode, implying an orthogonal arrangement of the irrelevant input directions with respect to the slowest left eigenvectors, as in the RNN mechanism. However, this difference in load across contexts resulted from very different mechanisms in the two models: in the *A, B ^{cx}* model it entirely reflects changes in the inputs, whereas in the

*A*model it entirely reflects changes in the recurrent dynamics. Unlike in the RNN mechanism, in both models the coherence input does not preferentially load onto the left-eigenvectors associated with the largest eigenvalues (|

^{cx}, B*λ*| > 0.9, time-constant

*τ*> 475ms). Rather, the largest loads are consistently obtained for eigenvectors with intermediate eigenvalues (|

*λ*| = 0.7 − 0.8,

*τ*= 140 − 224ms), and thus relatively fast decay time-constants (Fig. 5b,c).

### Non-normal dynamics makes model-specific contributions to selective integration

The qualitative similarities in the eigenspectra (Fig. 5b) and the input loads (Fig. 5c) of the *A, B ^{cx}* and

*A*models masks a key difference in the recurrent dynamics implemented by these models. Specifically, we find that the two models implement dynamics with very different degrees of non-normality. We assess the strength of non-normality through one of its possible consequences, namely the transient amplification of perturbations of the activity

^{cx}, B^{18,19}(Supplementary Note 2, Extended Data Fig. 7). We simulated the effect of injecting a short pulse of activity at trial onset, along random state-space directions. For the

*A, B*model, these perturbations gradually decay over the course of the trial (Fig. 6a, top, dashed lines, average across pulses in random directions). For the

^{cx}*A*model, on the other hand, activity following a pulse is transiently amplified, i.e. the gradual decay is preceded by a transient increase in activity (Fig. 6a, bottom, dashed lines). When perturbations are applied selectively along the left-eigenvectors, transient amplification is even more pronounced in the

^{cx}, B*A*model, but still largely absent in the

^{cx}, B*A, B*model (Fig. 6a, doted lines). Dynamics is thus strongly non-normal in the

^{cx}*A*model, as in the RNN mechanism, but less so in the

^{cx}, B*A, B*model (Fig. 6c).

^{cx}When dynamics is non-normal, the activity in response to a short pulse of input can reflect the combined effects of two processes operating at different time-scales: first, the transient, non-normal amplification of the input pulse; and second, its long-term integration towards a choice. The relative contributions from these processes differ across the two models, due to the different degree of non-normality of the underlying dynamics. In the *A, B ^{cx}* model, coherence input pulses are not transiently amplified, but rather immediately decay, whether they are relevant or not (Fig. 6b, top, thick doted and thin lines). In the

*A*model, the relevant input is transiently “persistent”, due to non-normal dynamics (Supplementary Note 2), whereas the irrelevant input quickly decays (bottom). Also at longer time-scales, the decay of a relevant input pulse is faster in the

^{cx}, B*A, B*model compared to the

^{cx}*A*model, indicating less accurate input integration. At both fast and slow time-scales, the recurrent dynamics of the

^{cx}, B*A, B*model thus cannot sustain information provided along relevant input dimensions as well as the dynamics in the the

^{cx}*A, B*model (Fig. 6b, top vs. bottom thick doted lines). As a consequence, the

^{cx}*A, B*model must instead rely on inputs that do not decay towards the end of the trial (Fig. 4b, top) to explain the observed persistent activity in PFC, whereas the

^{cx}*A*infers inputs that are more transient (Fig. 4c, top).

^{cx}, BThe pulse responses in Fig. 6b also illustrate that changes in the direction of the inputs across contexts are not sufficient to explain the differences in the representation of the relevant and irrelevant inputs in PFC. The employed pulses have unitary strength, and thus isolate the contribution of the input direction to the responses. The change in input direction across contexts in the *A, B ^{cx}* model lead to only relatively small differences in the amount of integration between the relevant and irrelevant inputs (compare late responses). To explain context-dependent integration in the PFC data, the

*A, B*model in addition must thus rely on contextual modulation of the strength of the inputs throughout the duration of the trial (Fig. 4b, top; relevant input is stronger than irrelevant input).

^{cx}The features of the dynamics considered so far imply that the two LDS models implemented mechanisms of selection and integration that share some key properties of the RNN mechanism. Like the RNN, all LDS models ultimately relied on a context-dependent realignment of the inputs and a subset of the modes of the recurrent dynamics, either through a change of the inputs (*A, B ^{cx}*) or of the recurrent dynamics (

*A*). Like the RNN mechanism, the

^{cx}, B*A*model (but not the

^{cx}, B*A, B*model) implements strongly non-normal recurrent dynamics. However, while the RNN mechanism relies on a single or few slow modes that are well aligned with the relevant input (an approximate “line attractor”

^{cx}^{1}), both LDS models instead implemented a large number of modes with different degrees of persistence, whereby the the inputs are not preferentially aligned with the slowest modes.

### Input integration occurs in two distinct phases

The above analyses provide insights into two mechanisms of context-dependent selection and integration that can account for population dynamics in PFC. However, these analyses alone do not explain how the neural trajectories predicted by the models emerge from the interaction of the inputs and the recurrent dynamics. Such an explanation must include also the properties of the right eigenvectors of the dynamics matrix, which amount to the “output” dimensions of the LDS models. The right eigenvectors influence both “where” in activity space the inputs are mapped onto and “how” they are transformed over time.

To establish how the trajectories emerge from the two LDS mechanisms, here we separately consider condition-dependent (CD) and condition-independent (CI) components of the neural trajectories. CD components were the primary focus of past accounts of this data^{1} and, particularly late in the trial, primarily capture choice-related activity. CI components, on the other hand, capture prominent structure in the neural responses that is related to the passage of time during a trial, and is common to all conditions and choices. To identify the modes of the dynamics that mostly account for CD or CI variance at a particular time in the trial, we computed the alignment between the right eigenvectors of the dynamics and the dimensions capturing most CD and CI variance at different times in the trial (Fig. 7, for the *A ^{cx}, B* model in the motion context, Extended Data Fig. 8, for all models and contexts). Only right eigenvectors that are well aligned with a given CD or CI dimension can contribute to the responses variance along that dimension.

The alignment between CD dimensions and right eigenvectors suggests that input integration occurs in two distinct phases, each dominated by distinct dynamics. Early in the trial, the CD responses occur primarily along right-eigenvectors corresponding to modes implementing relatively fast decay and fast rotations (|*λ*| = 0.7 − 0.8, decay time constant *τ* = 140 − 224ms, rotation frequency *f* > 1Hz, Fig. 7a,b, yellow lines). Late in the trial, the CD responses instead occur along right-eigenvectors with very slow decay and weak or no rotations (|*λ*| > 0.9, *τ* > 475ms, *f* < 0.25Hz, red lines). This transition was consistently observed across model classes (*A, B ^{cx}* and

*A*), contexts and model initializations (Extended Data Fig. 8). The differences in decay constants and rotational frequencies of the best aligned modes early vs. late in the trial are highly significant (Fig. 7b, Extended Data Fig. 8b, Wilcoxon rank-sum test, p<0.001). These observations imply that the relevant input is initially integrated along multiple decaying and rotational modes, consistent with the fact that the relevant inputs are strongly loaded onto left eigenvectors with intermediate eigenvalues (Fig. 5c). Later in the trial, the input is further integrated and maintained along at set of different, persistent and non-rotational modes.

^{cx}, BThe CI variance in the responses, on the other hand, appears to be mediated by largely different dynamic modes compared to the CD variance (Fig. 7c,d, Extended Data Fig. 8c,d). Unlike for the CD variance, the alignment between the leading CI direction and the right-eigenvectors is largely preserved across the trial. At all times in the trial, the leading CI variance occurs along directions associated with modes decaying more slowly than the early CD-aligned modes, but more quickly than the late CD-aligned modes (|*λ*| = 0.8 − 0.9, *τ* = 224 − 475ms). Likewise, these CI directions are associated with rotational frequencies that are smaller than those in early CD-aligned modes, but faster than late CD-aligned modes (*f* = 0.25 − 1Hz).

Overall, the inferred modes of the dynamics can thus be grouped into three distinct, non-overlapping sets, accounting for different components in the trajectories. The first and second set of modes account for early and late choice-related activity, while the third set accounts for choice-independent activity. The existence of these three different components of the PFC responses presumably explains why both models infer dynamics that is relatively high-dimensional and involving many modes associated with relatively slow decay.

To further validate the existence of multiple phases of the dynamics, we examined the activity trajectories along directions aligned with the relevant CD and CI components. Specifically, we defined an early and a late CD direction, which are primarily aligned with the first and second set of dynamics modes, respectively (yellow and red lines in Fig. 7a), and a single CI direction, which is primarily aligned with the third set of modes (green line in Fig. 7a), and then averaged these across contexts. To simplify the comparison with trajectories in Fig. 3, we projected the trajectories into two-dimensional subspaces spanned by the late CD direction and one of the other task-related directions.

We find that subspaces that include the early CD direction and the CI direction reveal prominent features in the population trajectories that are not apparent in other subspaces (Fig. 8a), confirming their potential importance in explaining the observed dynamics. The late CD direction closely matched the choice axis identified by Mante et al. (average angular difference of 18° across contexts, much more than expected by chance, Extended Data Fig. 6b) and captured a steady build-up of decision signals in both contexts over time^{1} (Fig. 8b, top panel, red dimension, dec). The early CD direction, on the other hand, captures an additional component of choice-related activity, which emerges early in the trial, but later decays (Fig. 8b, top panel, yellow dimension, dec 2). This decay is consistent with the above observation that early CD-directions are aligned with relatively fast decaying modes (Fig. 7b,c).

Together, the two CD directions thus capture decision-related activity evolving on different time-scales, whereby one component is transient and the other persistent. Notably, the projections along the early CD direction differ from those along the input directions (Fig. 8a,b, middle panels, black and blue mot and col dimensions, here the LDS-identified coherence input dimensions, averaged across models and contexts). Indeed, while activity along a given input dimension reflects the sign of a single input regardless of context, activity along the early CD dimension instead only reflects the sign of the contextually relevant input, and is not modulated by the irrelevant input (Fig. 8a,b, middle vs. top panels). Projections onto the CI direction likewise reveal additional components of the responses that are common to both choices (Fig. 8a,b, bottom panels). Finally, additional input-related dimensions can be defined based on the LDS fit, by considering variance due to coherence magnitude (|col| and |mot| in Fig. 4d,e, Extended Data Fig. 9a,b). All the inferred dimensions explained substantial fractions of the data variance (1-9%, Extended Data Fig. 9c) that are comparable to those captured by previously found task-related dimensions^{1} (Extended Data Fig. 9d).

Overall, these low-dimensional projections of the activity support the existence of the two phases of integration inferred from the analysis of the right eigenvectors. Moreover, these projections illustrate how the fits of the LDS models can be used to define a novel set of dimensions that appear to isolate the meaningful components of the computations implemented by the neural population.

### Trained RNNs do not capture all features of the PFC data

The properties of the inputs and dynamics inferred by the two LDS classes appear to differ in several ways from those expected by a simple line attractor mechanism, of the kind previously shown to be implemented by RNNs trained to solve the contextual integration task. In particular, both LDS models implement a decision process that unfolds in two phases (early vs. late choice dimensions), rely on the contextual modulation of a large number of dynamics modes across a wide range of decay constants, and infer multi-dimensional inputs modulated both by signed and unsigned input coherence. However, it is not immediately clear that these features reflect meaningful differences between the mechanisms implemented by the LDS models and the RNN. While the RNN are non-linear, the LDS are linear, meaning that some LDS features may simply reflect somewhat trivial consequences of approximating non-linear dynamics with a linear system.

To evaluate this possibility, we repeated all the analyses we performed on the PFC responses also on simulated responses of a trained RNN, and then directly compared the two (Supplementary Figs. 1 to 5). This comparison shows that the features of the PFC responses highlighted above are not captured by the trained RNN. First, contextual modulation of the dynamics in the RNN is most pronounced in modes that are persistent or slowly decaying (Supplementary Fig. 3b), whereas in PFC it is strongest in relatively quickly decaying modes (Fig. 5c). In the RNN, as in PFC, the inferred slow dynamics is not limited to a single mode, unlike in a perfect line attractor (Supplementary Fig. 3a and Fig. 5b). This observation is expected, as the RNN tend to implement integration along a one-dimensional manifold that is curved, rather than perfectly straight, and thus cannot be approximated by a single linear mode. Second, the LDS fits do not provide any evidence of multiple phases of input integration in the RNN (Supplementary Fig. 4), whereas they reveal distinct early and late phases in the PFC responses (Extended Data Fig. 8). Finally, the LDS fits of the RNN responses learn inputs that are largely one-dimensional, whereby input signals are modulated by absolute input strength only weakly or not at all (Supplementary Fig. 2c-e). This finding contrasts with the robust encoding of absolute input strength in PFC (Fig. 4d,e).

The analyses of the RNN responses also reiterate the challenges in establishing which of the two mechanisms implemented by the LDS models is more likely to be implemented by PFC. Indeed, as for the PFC data, also the RNN data can be well fit by both model classes (Supplementary Fig. 1), even though arguably only one of the two classes matches the RNN in how it selects and integrates the sensory inputs. In the trained RNN, the inputs are not modulated by context, and input-related responses in the network are likewise largely constant across contexts. This property matches the design of the *A ^{cx}, B* model, but not the

*A, B*models, and yet both models fit the RNN data equally well. However, the

^{cx}*A, B*fits of the RNN responses do display some idiosyncratic properties suggestive of a very precise fine-tuning of parameters. In particular, the

^{cx}*A, B*model required many more dimensions than the

^{cx}*A*model to fit the data (Supplementary Table 5) and presented extreme levels of amplification for dimensions other than the input dimensions (Supplementary Fig. 3c-e). Such a fine-tuning may reflect the mismatch between the underlying mechanisms of integration. Notably, we did not find such evidence of precise fine-tuning in the fits of the PFC responses, meaning that also in this respect both model classes are equally valid descriptions of the PFC data.

^{cx}, B## Discussion

The complex and highly heterogeneous activity patterns observed in prefrontal areas are thought to reflect the specific computations implemented in these regions^{20}. In this study, we inferred candidate mechanisms for one of such computation, contextual decision-making, directly from PFC responses. By fitting several LDS models to the PFC data, we inferred interpretable dynamics that linearly approximate the neural activity in each context. We found that two distinct mechanism of contextual-integration were consistent with the PFC activity: a switch in recurrent dynamics and a modulation of inputs. The key features of these mechanisms were consistently found across the motion and color inputs in monkey A, and the motion input in monkey F (Supplementary Figs. 6 to 10). As previously reported^{12}, representations of color inputs were instead weak or absent in monkey F.

The first LDS mechanism is broadly consistent with past accounts of PFC responses in this task^{1}, in that the input selection relies on non-normal, context-dependent recurrent dynamics. In addition to this role in inputs selection, our analysis revealed that non-normal dynamics might additionally result in the transient amplification of relevant inputs in PFC. Non-normal transient amplification had previously been proposed to play a role in the processing of external inputs^{21–23}, as well as computations as varied as maintaining inputs in working-memory^{24}, generating the transient neural activations required for generating movements^{25} and mediating robustness to perturbations^{26}. Our observation of two distinct stages in PFC dynamics during decision formation is evocative of a recently proposed mechanisms relying on transient amplification to optimally load information onto an attractor^{22}. In contrast to the predictions of such optimal loading, however, we found that the inputs where not preferentially aligned with the most amplifying dimensions of the dynamics (Extended Data Fig. 6e-g).

The second LDS mechanism relies on modulation of the inputs, and could be implemented via top-down influences on sensory areas. Our LDS fits reveal how strong such top-down modulation would have to be to explain context-dependent responses in PFC. The inferred modulation strengths (38 ± 14% mot, 22 ± 8% col, Fig. 4b) are in the range of some attentional effects observed in sensory areas^{27,28}, although other studies have reported substantially weaker or stronger feature-based modulation^{29–33}, potentially reflecting differences in task design, cognitive demands, sensory features, and areas. In particular, our findings differ from recent modeling of sensory and prefrontal responses during auditory, contextual-decision making^{33}, in that the irrelevant input is not completely gated out before reaching PFC, and the relevant input is integrated entirely within PFC. Notably, we inferred not just a modulation of the input strength, but also of its direction. Such a change in direction could be achieved with top-down modulation if each input was multi-dimensional (i.e. originated in multiple neural populations or areas; Supplementary Note 3) and individual dimensions were modulated independently. Alternatively, input amplitude and direction could both be modulated by non-linear dynamics occurring within PFC,^{34}, a possibility that we did not explicitly model here.

Our fits revealed novel features of the population responses in PFC that were consistent across both LDS mechanisms. Both models implemented input integration in two distinct phases, whereby choice-related signals first emerged along relatively fast decaying dimensions with rotational dynamics, and then transitioned towards orthogonal dimensions with slower, non-rotational dynamics. As a result, individual task-related signals were encoded dynamically along multiple dimensions at different time-scales, consistent with previous analyses of this data^{12}. Beyond describing the structure of these signals, here we show how they could emerge dynamically from the interaction of inputs and recurrent dynamics. The dynamics we inferred differs in several ways from that implemented by previously proposed one-dimensional line-attractors^{1,16}. Nonetheless, in agreement with such simpler models, we found that at longer time-scales decision signals emerged predominantly along a single integration dimension that was common across contexts^{1} (Extended Data Fig. 10).

The LDS models provide several insights into the properties of potential inputs into PFC, beyond their contextual modulation. Both mechanisms inferred multi-dimensional inputs carrying information about both signed coherence and coherence magnitude. As a result, the inputs defined curved manifolds with respect to coherence, in agreement with findings in parietal and frontal areas^{11,12,15}. Our results strengthen these previous findings, as in our models the different input components were inferred entirely from the data, rather than being hand-designed^{12}. Second, both models inferred inputs that were somewhat transient, even though the fits penalized large magnitude inputs. The fits inferred inputs that became weaker (*A, B ^{cx}* mechanism) or progressively decayed (

*A*mechanism) late in the trial (Fig. 4b,c). However, models with time-invariant inputs cannot be ruled out, as they resulted in comparable performance (Extended Data Fig. 5b) and captured the neural trajectories nearly as well (Extended Data Fig. 2g,h). Critically, these models relied on mechanisms that were analogous to those described above (data not shown), confirming that the complexity of the PFC responses is well approximated by linear dynamics and not necessarily inherited from inputs with rich dynamics.

^{cx}, BOur models provide an alternative to previously proposed approaches to inferring the properties of inputs into an area. One advantage over past approaches^{35,36} is that we make minimal assumption about the properties of the inputs, and in particular about their dimensionality. While several studies have emphasized the importance of inferring inputs to understand cortical dynamics and function^{11,35,37–39} such efforts are complicated by unavoidable model degeneracies that arise when attempting to distinguish inputs from recurrent contributions without access to the upstream areas from which the inputs originate^{35,40,41}. Our finding that two fundamentally different mechanisms of input selection explain PFC responses equally well is a reflection of such degeneracy. Indeed, both the inputs and the choice-related signals inferred from PFC activity may reflect computations distributed across several cortical areas^{2}.

Our modeling approach has the advantage of decomposing the dynamics of a complex system into simpler linear parts that are amenable to analysis and interpretation, similar to switching LDS models^{42}. A previous application of such models has led to the discovery of line attractor dynamics in the hypothalamus of mice during decisions underlying aggression^{43}. In combination with methods from control theory, LDS can also be used to infer inputs that are optimal for a given task, like bringing brain activity into healthy regimes in biomedical applications^{44} or optimally configuring cortical dynamics during movement preparation^{37–39}. Here, we found that our fitted LDS models are fully controllable^{37} (data not shown), and applied methods from control theory to identify the most amplifying dimensions of the dynamics^{22}, but an exhaustive analysis of this type is beyond the scope of our study.

We validated the assumption of linear dynamics by comparing the LDS fits to the fits from our novel TFR model. The LDS models explained the data essentially as well as TFR, which sets an upper bound to the goodness of fit achievable by an LDS. The success of the LDS models imply that, in PFC, intuitive linear descriptions apply to all regions of state space, and not only to local regions around fixed points^{1}. While here we fitted activity from only a relatively short time window from each trial (the 750ms of random dots presentation), recent findings suggests that linear models may not be outperformed by non-linear models in capturing cortical dynamics even over longer time-windows^{45}. Nonetheless, analyses based on non-linear models are becoming increasingly common, given their flexibility in capturing very complex neural data^{35} and the interest in modeling biological constraints that cannot be captured by linear models^{46} (but see^{26}).

A crucial aspect of our data-driven modeling approach is that we tested multiple model designs corresponding to specific computations underlying the measured activity. No single model can be expected to perfectly explain the rich dynamics observed in areas like PFC. Thus, it is important to test multiple alternatives hypotheses and identify all models that are plausible explanations of the data, rather than committing to the best model as the “correct” one^{14,47}. Indeed, we found that several LDS mechanisms explained the data similarly well (*A, B ^{cx}* and

*A*models with time-varying 3D inputs, Fig. 2a, Fig. 3c,d, and 2D inputs Extended Data Fig. 2c,d; and time-constant 3D inputs, Extended Data Fig. 5b, Extended Data Fig. 2g,h), whereas others explained the data less well (models with time-varying 1D inputs, Fig. 2a, Extended Data Fig. 2e,f) or only poorly (a

^{cx}, B*A, B*model, fully constrained across contexts, with time-varying 3D inputs, Fig. 2a, Extended Data Fig. 2b). The mechanisms we identified as plausible explanations of the PFC responses share key features with mechanisms of context-dependent integration that were recently described in rats

^{48}. Notably, that study demonstrated the advantage of pulsatile inputs in distinguishing between different mechanisms of input selection and integration. Similarly, one approach for distinguishing between the two candidate mechanisms we identified would rely on studying the dynamics following perturbations along random state-space directions, which would evolve differently under the two mechanisms due to the different degree of non-normality in the dynamics (Fig. 6a). Alternatively, in simultaneous recordings from large groups of neurons, input and recurrent contributions to the dynamics may sometimes be distinguished based on the properties of trial-by-trial variability of the population responses

^{40}.

Methods for inferring neural population dynamics of the kind proposed here will likely play a key role in uncovering the neural computations underlying behavior. While abstract mental processes were originally hypothesized to reflect structural changes at the level of single neurons (Santiago Ramón y Cajal, see^{49}), more recent evidence suggest that cognitive functions arise at the neural population level and depend critically on the ability of neural circuits to flexibly switch between dynamical regimes^{17,50–52}. Ultimately, a complete description of neural computations will also explain how neural dynamics emerges from the rich and dynamic structural components of biological circuits^{53–55}. The lawful characterization of population level dynamics amounts to a theoretical abstraction of the neural computations emerging from such a rich neural circuit, and provides a key bridge in linking lower-level biological structure to behavior.

## End Notes

## Author Contributions

J.S.M. and M.S. conceptualized the methodological approach, developed the models and performed data analysis. V.M. conceived the experiments, collected the data and conceptualized data analysis. All authors actively participated in the interpretation of the data. J.S.M and V.M wrote the paper.

## Code Availability

The code will be made publicly available on GitHub upon peer-reviewed publication.

## 1 Methods

### 1.1 Experimental procedures and data

#### 1.1.1 Subjects and task

Two adult male rhesus monkeys were trained in a contextual two-alternative forced-choice visual discrimination task. The monkeys had to discriminate either the color or the motion of a random dots display based on context, which was indicated by the fixation cue. The presentation of the random dots lasted for 750ms, after which the monkeys had to wait for a variable delay and report their decision. This was done by saccading to one of two diametrically opposite targets, as indicated by the color or motion evidence. The strength of the evidence was modified by varying the motion and color coherence of the random dots. This was determined by the percentage of dots moving coherently or that were colored the same. Six different coherence settings were used: three strength levels and two directions. The later indicated whether the evidence was pointing towards or away from one of two choice targets—placed at the receptive field (RF) location of the recorded neurons. When the evidence pointed towards the RF of the neurons, their FRs typically increased above baseline. Therefore, positive values were used to define the in-RF evidence. On the contrary, when the evidence pointed away from the RF of neurons, their FRs typically decreased, and hence negative values were used to define the out-RF evidence. Considering all possible motion and color coherence value pairings (6×6), 36 different random dots configurations were presented, which defined the 36 task conditions. Importantly, the motion and color evidence in a given trial could be congruent or incongruent. When incongruent, it was necessary for the monkey to ignore the irrelevant signals in order to perform the correct decision. For further details on the animal procedures and task we refer to the original study^{1}.

#### 1.1.2 Neural data

Electrophysiological recordings were performed during the task in PFC regions, likely comprising the frontal eye fields (FEF) and surroundings. Both single-unit and multi-unit activity was isolated from the recordings. We referred to them as neurons, for simplicity. Only a few neurons were recorded simultaneously in each trial, but their activity was collected for multiple trials under the 36 different task conditions. Population responses were then constructed by pooling the condition-averaged activity of all neurons. For that, the firing rate of the neurons was computed in each trial using a 50ms sliding square window from spike trains sampled at 1ms. Activity was then averaged across trials under the same condition and z-scored, as in^{1}. However, we did not apply any smoothing to the data prior to fitting the models (only in the analysis, for visualization purposes). Thus, the data consisted of a pseudo-population of per-condition averaged PSTHs. The population size was N=727 for monkey A and N=574 for monkey F. We included only neurons which had recorded activity under all conditions and for all times. As in the original study, we focused our analysis on the period of random dots presentation (750ms, from 100ms after dots onset to 100ms after dots offset) and we analysed only correct trials.

### 1.2 Models

#### 1.2.1 Linear Dynamical System (LDS) model

The linear dynamical system model considered was a non-probabilistic version of a standard LDS or state space model, with equations
where the vector **x**_{k}(*t*) represents the latent state at time step *t* and task condition *k*, **y**_{k}(*t*) are the observations (a vector containing the PFC condition-averaged PSTHs) and **u**_{k}(*t*) the external input vector. The dynamics matrix *A* determines the transition between subsequent latent states. The matrix *B* defines the input dimensions. The external inputs drive the dynamical system at each time step and define input vectors (*B***u**(*t*)) that live in the latent subspace spanned by the columns of *B*. Therefore, the external inputs are assumed linearly mixed in the population at each time step. Note that the input vectors (*B***u**(*t*)) can point in different directions over time, but these changes are always confined within the input subspaces. The input term in equation (1) can be decomposed to make explicit its color and motion components *B _{m}*

**u**

_{m}(

*t*) +

*B*

_{c}**u**

_{c}(

*t*). The loading matrix

*C*maps the low-dimensional latent state onto the high-dimensional neural space. The constant vector

**d**acts as a bias. This LDS model can be seen as a low-dimensional RNN that reads-out onto a high-dimensional output space.

To capture changes in activity across contexts , we fitted an LDS model jointly to the PFC data from each context. The model could learn independent parameters for each context (based on the data from each context) or a single parameter across contexts (using the joint data from both contexts). Both the dynamics matrix *A ^{cx}* and the motion and color subspaces could be context-dependent (

*cx*= mot or col context). The matrices could have different norms, and hence, contextual modulation of inputs could be implemented through changes in both input subspace orientation and norm. The external input signals

**u**

_{m,c}(

*t*) and the mapping

*C*were assumed fixed across contexts.

For each motion and color input dimension, 6 external input time courses were learned, corresponding to the 6 different coherence values in the task (3 strength levels and 2 directions). These were inferred pooling data from all task conditions were a particular coherence level was presented, and therefore, were shared across task conditions (36 task conditions). The model incorporated additional input constraints, which simplified their temporal structure and were found to improve generalization performance. Time courses were constrained to be the same for all coherence levels of the same direction. That is, a single time course was shared for positive coherences (in-RF evidence) and another one for negative coherences (out-RF evidence). The coherence strength level was learned as a scalar value that multiplied the time course *u*(*t*) = *T _{in,out}*(

*t*)

*coh*

_{1,…,6}. We also fitted a model constrained to learn fixed inputs in time, with

*T*(1, …,

_{in,out}*t*) = 1. The resulting input vectors (

*B*

**u**) also lived in the input subspace defined by the input matrices

*B*, but unlike the input vectors for the time varying input model (

*B*

**u**(

*t*)), these do not move within the input subspaces over time, and remain fixed throughout the trial both in strength and direction.

A different vector of initial conditions was also learned for each context . This parameter helped the model recreate the separation of trajectories in state space found across contexts (contextual axis in the Mante et al. study^{1}). Note that this feature cannot account for the contextual differences in input integration, since the model is linear, so the relationship between inputs and dynamics modes is the same everywhere in state space. Indeed, a fully constrained model across contexts, with flexibility only in the initial conditions, fails to selectively integrate and poorly reproduces the data (Fig. 2a *A, B* model, Extended Data Fig. 2b). The initial conditions simply add a shift to the overall dynamics in an input independent manner, since **x**_{0} is the same across all task conditions, so it could only capture baseline changes across contexts. This can be seen in the next equation, which illustrates the unfolding of the dynamics from the initial state and makes the dynamics and inputs convolution explicit

This equation also illustrates the presence of a summation degeneracy in the model. The first term defines condition independent (CI) effects, but these can also be captured by the input term. For this reason, in figure Fig. 4, Fig. 8, Extended Data Fig. 3b, Extended Data Fig. 4, Extended Data Fig. 9 and associated Supplementary Figures, we subtracted out CI effects from the input/data trajectories along the input dimensions.

The model was implemented in Python and optimized using gradient descent (ADAM algorithm) to minimize the data reconstruction mean squared error (MSE), with learning rate of 0.009, and the rest of parameters set to the default. The convergence criteria was set to ΔMSE < 10^{−5}, maximum iterations to 10,000 and minimum iterations to 5,000. The cost function incorporated an input norm penalty to constrain the space of possible solutions and to favour learning small inputs. This encouraged that task-related variables in the data other than the inputs, in particular integration signals, were generated dynamically by the model. Incorporating the penalty minimally impacted performance and helped provide consistent solutions across fits even when parameters were initialized at random. Therefore, we incorporated such penalty in all our model fits and randomly initialized all parameters. The resulting objective function was:

Where N = number of neurons, T = trial duration, K = number of conditions and C = number of contexts. Since the data was z-scored, the MSE captured the fraction of unexplained variance in the data by the model.

Note that the LDS was simply optimized to minimize the MSE of the condition-averaged PSTHs. We did not learn any observations noise model or inferred a latent state distribution, contrary to more standard formulations of the LDS, which are fully probabilistic (and typically infer Gaussian latents, or Gaussian latents combined with Poisson observations^{56}). We considered this simpler case given that our data was trial-averaged. Furthermore, our focus was to analyse the parameters of the dynamical model, which are part of the prior distribution over the latents in the probabilistic LDS, and not the data-corrected posterior distribution.

#### 1.2.2 Tensor Factor Regression (TFR) model

The model consists of a factorization of the data tensor structure into three main low-rank tensors
where *n* = number of neurons, *t* = time steps, *k* = conditions, *l* = latent dimensionality, *u* = input dimensionality + 1D baseline. The tensor *C* (an orthonormal matrix) sets the rank of the factorization and maps the low-dimensional core tensor *AB* into the high-dimensional neural space. The inputs tensor *U* captures the condition-dependent effects in the data and acts as a regressor, when this is known. When learned, as it is the case here, it is used to capture task-related variables, such as motion and color input signals. Note that similar to the LDS, these signals are assumed linearly mixed in the population at each time step.

In the previous equation, for clarity (as in Fig. 2c), we omitted an indicator tensor *T* that emulates the LDS-like convolution of inputs and dynamics
where *T _{tt′t″}* =

*δ*(

*t*−

*t*′ =

*t*″). One can see how this model encompasses the LDS by writing where

*A*and

*B*correspond to the LDS dynamics and input subspace matrices, respectively.

The inputs incorporated constraints analogous to the LDS. First, inputs were repeated across conditions with an additional indicator tensor *Q*
where *c* = (6 × *u*) + 1 indexes the six coherence conditions, plus baseline (that captures CI effects). In this way, the tensor *U* is designed to extract common task-related variables across conditions. Second, the temporal structure of the inputs was constrained to be the same for coherences of the same direction. For that, the input tensor *U* was factorized further as follows
where *d* = 2 indexed the two possible coherence directions.

The parameters of the TFR model can be computed by alternating the estimation of the tensors *W* = *CAB* and *U*. For that, one can consider the tensor unfolding *Y*_{(n)(tk)} and compute *C* and *AB* via reduced rank regression, with fixed *U*. Then, knowing *W*, the least squares estimate of *U* can be computed. In practice, we estimated the parameters following the same optimization procedure we used for the LDS, which provided identical results. That is, the model was implemented in Python and optimized using ADAM, with objective given by the data reconstruction MSE.

The TFR model is related to existing regression-based methods that discover task-related variance in the data^{1,12,57}, but with the difference that TFR incorporates task regressors that are themselves learned from the data. Another key distinction is that TFR considers a joint factorization of the whole data tensor structure, similar to other studies^{58}, but the tensor components relate to the parameters of the task and are themselves low-dimensional.

#### 1.2.3 Recurrent Neural Network (RNN) model

We generated data from a RNN model of the same type as used by Mante and colleagues^{1}.

Briefly, the model was a non-linear RNN trained using back-propagation to solve the same contextual decision-making task as the monkeys. Contrary to the LDS, the RNN was not optimized to reproduce the complex and heterogeneous responses of PFC neurons, i.e. to match PFC’s dynamics. This network was designed with the same built-in assumptions as in the original model (Fig. 1c). Namely, that the external coherence input signals *u _{m}* and

*u*were noisy but constant in time, with mean proportional to the strength of the coherence evidence, and that these reached the circuit through two fixed input dimensions across contexts

_{c}**b**

_{m}and

**b**

_{c}. The model had the flexibility to learn different contextual input vectors

**b**

^{cx}, whose activation changed the dynamics of a fixed, non-linear recurrent network (with connectivity

*A*). This allowed the model to switch its state between two approximately linear regimes , performing different computations in each context. Namely, selecting the contextually-relevant input signals for integration towards a choice and dynamically discarding the irrelevant ones. In the original study, the RNN population activity was analysed and qualitatively compared with the PFC activity, revealing some shared features that were suggestive of a common contextual-integration mechanism between PFC and the network. The network could be “reverse-engineered” in order to understand the mechanism underlying such computation, by linearizing the dynamics around the identified fixed points of the system (obtaining different local , which however were similar in dynamics and could be averaged

^{1}). In this work, we instead focused on analysing the properties of LDS models fit to the RNN population activity (the z-scored condition-averaged responses, as in the PFC data, from a 100 RNN units). For further details on the RNN training and analysis we refer to the original study

^{1}.

### 1.3 Dynamics analysis

#### 1.3.1 Eigenspectrum and time constants

The eigenspectrum of the dynamics matrix contains both real and imaginary eigenvalues, which come in complex-conjugate pairs

The absolute value of the eigenvalues determines the rate of decay or growth of each dynamic mode^{59}. Modes are stable if they either decay or persist

The slower the decay, the slower or more persistent a given mode is, and the greater input information is preserved along it. The time constant measures the time at which the initial state will have decayed by 37% (1/e=0.37) along a given mode. Considering that each time step is 50ms (the data binning size)

We consider that a mode is slow if it has a norm close to one, that is, if |*λ*| > 0.8. This corresponds to a decay time constant of *τ* > 224ms, which encompasses approximately a third of the trial duration. Given that the inferred external inputs in the two models are strong for the first third of the trial (Fig. 4a,b), inputs mapped onto such slow modes largely persist until the end of the trial, albeit with some decay for modes |*λ*| = 0.8 − 0.9. In particular, by the second third of the trial, inputs would have decayed by at most 37%. We consider the slowest modes to have |*λ*| > 0.9 and time constant *τ* > 475ms. These are strongly persistent and preserve most input information until the end of the trial. The relatively fast decaying modes (|*λ*| = 0.7 − 0.8, *τ* = 140 − 224ms) are somewhat persistent, but loose most input information by the end of the trial.

Many of the eigenvalues were imaginary, indicating the presence of rotational dynamics in the data^{17}. Some of the eigenvalues were negative, which also indicate the presence of oscillations^{44}. A few models identified slightly unstable eigenmodes (with eigenvalue norm slightly bigger than 1), but this is expected when learning from finite trial lengths and limited data samples^{60}. However, the models inferred from monkey F data, in particular for the *A, B ^{cx}* model, seemed to use instability properties of the dynamics in order to capture specific features of the data (Supplementary Fig. 8a,d).

#### 1.3.2 Rotational dynamics measure

The existence of complex eigenvalues indicates the presence of rotational dynamics in the data. Rotations are confined to the planes defined by pairs of complex-conjugate eigenvectors, with directions spanned by the real and imaginary components of the vectors^{59}. Considering first the phase plane representation of a complex-conjugate pair of eigenvalues in polar coordinates
where

Rotations on each plane are determined by the rotation matrix *J*, which derives from the dynamics matrix expressed in the Jordan normal form^{59}. As an example, for a 2D system with two distinct complex eigenvalues

Rotations evolve in time following powers of *J*, with amplitude over time (the rate of decay or growth) given by the absolute value of the eigenvalues

Therefore, *ω* is the rotation frequency in the complex plane. Note that the frequency increases when the ratio is big. The rotation frequency *ω* is given in *rad/s* and *f* = *ω*/(2*π*) in Hz. Since the data was down-sampled at 20Hz (50ms bins), the frequency is given by *f* = 20*ω*/(2*π*) in Hz. For real modes, the rotation frequency is zero.

#### 1.3.3 Non-normality measure

The Henrici’s index measures the degree of non-normality of the the dynamics, and is given by^{61}

This is a normalized metric with values between 0 and 1, with 0 indicating that the system is normal, and 1 that is maximally non-normal. A system is normal when its dynamics can be described with an orthonormal eigenvector basis. A system is non-normal when its eigenvectors do not necessarily form an orthonormal basis, and the transformation to eigenvector coordinates may involve a strong distortion of the phase space^{61}. Importantly, in normal linear networks, the network responses are explained with a linear combination of exponentially decaying modes (if the system is stable), with timescales defined by the corresponding eigenvalue (equation (12)). In non-normal stable networks, however, more complex patterns can emerge, which often involve transient responses where the network activity temporarily grows, but eventually decays as in normal systems.

A crucial property of non-normal systems is that they have different left and right eigenvectors.
with *L* = *R*^{−1}, whereas for normal systems *L* = *R*^{†} (†=conjugate transpose). This non-normal property allowed the RNN to change the leading left eigenvectors across contexts, while keeping the right eigenvectors pointing in the same direction.

#### 1.3.4 Input loads

To compute the input load onto the modes of the dynamics, we start by expressing the latents in the left eigenvectors basis.
where we have taken the eigendecomposition of the matrix *A*, with *R* containing the right eigenvectors in its columns and *L* = *R*^{−1} the left eigenvectors in its rows. We have then left-multiplied by *L*. Defining ** α**(

*t*) =

*L*

**x**(

*t*) we obtain

The evolution of the latents in this basis is independent, that is, decoupled from one another—given that the matrix Λ is diagonal. Unrolling this equation in time we obtain

As the eigenmodes are independent, we can write down a set of uncoupled equations that describe the evolution of each eigenmode, one for each entry of the vector ** α**, given by

*α*with l indexing the latent variable dimension and

_{l}

*l*_{l}being the lth left eigenvector. The input “loads” are defined by the last term of the summation, which correspond to the non-normalized projection of the inputs onto the left eigenvectors (note that neither the input vectors nor the left eigenvectors are unit norm).

This term specifies how strongly the inputs are mapped onto the dynamic modes, at each time step *t*, before being processed by the dynamics (i.e., in this basis, before being scaled by *λ*). The extend to which the inputs are mapped or “loaded” onto each mode depends on the alignment between the input vectors and each left eigenvector, as well as the norm of both vectors. For each pair of complex modes, the load is given by

Where *ℜ*{.} and *ℑ*{.} take the real and imaginary components of their arguments. The rationale for the expression above comes from the following. For complex modes, equation (23) contains imaginary numbers, since the left eigenvectors are complex, so we cannot interpret the loads in this basis. However, we can do it in the original state vector basis **x**(t), which is real. To change basis, we use ** α**(

*t*) =

*L*

**x**(

*t*) and express

**x**(

*t*) as a linear combination of the state along each right eigenvector dimension. The coefficients of the linear combination are given by

*α*(

_{l}*t*), which contains the input loads

We can now make explicit the contribution due to real eigenmodes and complex eigenmodes, which come in complex conjugate pairs (*l − l*^{†}).

Due to the complex conjugacy, the imaginary numbers end up cancelling out in the summation, and only real terms survive. This is why in this basis, the state vector **x**(t) is real. In particular, the way the complex roots end up contributing to the state dynamics is given by their real and imaginary parts. This is because for each pair of complex conjugate roots, two complementary real solutions exist, which are given by the sum and difference modes *α _{l}*± (

*t*)

This can be seen by expanding the complex term in the state equation

Thus

To understand how the inputs are loaded at each time step *t* into the dynamic modes to affect the latent state, we focus on the last term of the summation in the *α _{l}*(

*t*) equation (22), as we did before

The last term contains the input loads along each real mode, , which gives equation (23). This value indicates how much of the input is mapped along each right eigenvector direction *r*_{l}. Thus, considering only this term, the latent state vector is reconstructed with a linear combination of real right eigenvectors, weighted by the input loads. Note however, that the right eigenvectors are not orthogonal, so the result of the sum could be non-trivial, if for instance some of this vectors cancel out, or give rise to amplification (Supplementary Notes, Extended Data Fig. 7). The total input contribution or load along each direction *r*_{l} is thus given by the norm of the vector . Since the real right eigenvectors are normalized, this is equal to , which gives equation (23). Similarly, the load for each complex conjugate pair of modes is given by the norm of the vector , which gives equation (24). This vector lives within the 2D plane spanned by the real and imaginary components of the complex-conjugate right eigenvector pairs.

To compute the loads in Fig. 5c, we use the inferred inputs for the largest motion and color positive coherence values, and project them along the coherence dimension. So the loads are computed using the coherence component of *B***u**(*t*), for all times and all 100 models, and then averaged across time and models. For complex modes, the same load is shared across both complex conjugate pairs, and is computed using equation (24).

#### 1.3.5 Most amplifying dimensions

The most amplifying modes where found following^{22}, by computing the Observability Gramian and its associated eigenvectors. The most amplifying modes are defined by the eigenvectors with the largest associated eigenvalues. We computed the Observability Gramian by solving the following Lyapunov equation
where A is the LDS models dynamics matrix and C is the loading matrix. We considered only stable models^{22}, which in our case were 90% of the 100 *A, B ^{cx}* models and 85% (mot cx), 60% (col cx) of the

*A*models in monkey A.

^{cx}, B### 1.4 Additional analysis methods

#### 1.4.1 Alignment metrics

We report alignments between different dimensions using either dot products or angles (in degrees). When computing alignments between a given vector and complex eigenvector dimensions, we consider the plane spanned by the real and imaginary vector components of the pair of complex conjugate modes, and compute the minimum subspace angle between the vector and the plane.

#### 1.4.2 Statistical tests

To asses statistical significance of differences between distributions, such as the relevant vs. irrelevant load distributions in Fig. 5c, we used a Wilcoxon rank-sum test with significance levels (p-values) set at p<0.001 (Fig. 4b,c, Fig. 7b, Extended Data Fig. 8b, and associated Supplementary Figures) or p<0.05 (Fig. 5c and associated Supplementary Figures, also Supplementary Fig. 4). This is a two-sided rank sum test of the null hypothesis that two independent samples come from distributions with equal medians.

## Supplementary Information

### 1 Supplementary Notes

#### 1.1 Additional LDS model-fitting controls

To rule out the possibility that the LDS models were learning dynamically complex external input signals **u**_{m,c}(*t*) to capture the data, we re-trained all input parameters and constrained **u**_{m,c}(*t*) to be constant in time **u**_{m,c}. This resulted in a relatively small drop in performance (Extended Data Fig. 5a, for models with input dimensionalities higher than 2D). In particular, the performance of re-trained models with 3D constant inputs (dashed lines) dropped to the level of the 1D time-varying input models (filled lines). Newly fitted models with **u**_{m,c} constant in time, but where all parameters were optimized, performed nearly as well as the time-varying models (Extended Data Fig. 5b), and accurately captured the PFC trajectories (Extended Data Fig. 2g,h). Notably, in these two control model classes the optimal input dimensionality was consistently 3D, and the latent dimensionalty was also close to the time-varying input models’ dimensionality (*A ^{cx}, B* -

*A, B*xdim = 16-22 for the time-varying models with inputs retrained to be time-constant; 15-16 for the newly optimized time-constant models and xdim=16-18 for the original time-varying models). Most importantly, the time-constant input models could only rely on their recurrent dynamics to capture the temporal complexity of the PFC data. Therefore, the complexity of the PFC responses is well approximated by linear dynamics and is not necessarily inherited from the external inputs’ dynamics.

^{cx}We also quantitatively assessed whether the time-constant input models were indeed capturing any temporal structure in the data. For this, we asked how well these models performed on time-shuffled data, which had no correlational structure across time. If there was no drop in performance, this would indicate that the time-constant input models were uniquely capturing time-unrelated correlational structure, such as correlations across neurons and conditions. The performance of the best time-constant models (which had 3D inputs, Extended Data Fig. 5b) dropped substantially, being worse than the 1D input models and nearly as bad as the most contextually constrained *A, B* models (Extended Data Fig. 5c). This indicates that the simple, time-constant LDS models were indeed capturing time-related structure in the PFC data. However, these models still captured a substantial fraction of the time-shuffled data variance (24% on shuffled data vs 27% on the original data). This suggests that the LDS models might be primarily capturing correlational structure across neurons and conditions, besides time-related variance. Indeed, surrogate data sets randomized across conditions, neurons and time were very poorly captured, even by the best time-varying LDS models (Extended Data Fig. 5d). These data sets were designed to preserve the primary statistics of the data, and thus these results also indicate that the LDS models were not merely capturing basic features of the data^{13}.

#### 1.2 Understanding non-normal transient amplification

Transient amplification is a property of dynamical systems that have non-normal dynamics matrices^{18,19}. These systems present non-trivial dynamical properties that are not predicted by their steady state behavior. In particular, such systems can transiently amplify inputs before decaying to a steady state. To illustrate how the transient amplification mechanism takes place, we built two simplified dynamical system models with identical specifications and only two dimensions, and made one normal (Extended Data Fig. 7a, degree of non-normality or Henrici’s index=0, see Methods) and the other highly non-normal (Extended Data Fig. 7b, Henrici’s index=0.8). In Extended Data Fig. 7a,b, we show the two right eigenvectors of the dynamics (*e*1, *e*2, in green and blue), for each system, which define the “output” dimensions were activity evolves over time. In the normal system, by definition, these are orthogonal vectors. In the non-normal system, they need not be, as it is the case in this example. Two additional left eigenvectors exist, but in a normal system they are the same as the right eigenvectors (see Methods). In a non-normal system the left eigenvectors are distinct from the right ones (see Methods), but these are not shown here since they are not crucial to understand this picture. We set one of the eigenvalues of the system to be small (*λ* = 0.3, fast dynamics) and the other large (*λ* = 0.7, slow dynamics) in both models. We then analysed the impulse response properties of both systems (for t=15 time steps, as in the PFC data). For this, we provided an input pulse of unit norm along a random direction *x*0 (in this case, for illustration purposes, a direction that bisected the plane spanned by the two right eigenvectors in the normal system, left panels in Extended Data Fig. 7a,b, in purple). We then looked at how each system processed such input by looking at the evolution of the system state over time *x*(*t*) and its norm (middle and right panels, in purple). In the normal system, the input decayed exponentially towards its steady-state, at zero, as expected from a dynamical system with real eigenvalues smaller than one (right and middle panels). The evolution of the two modes of the dynamics was governed by the time-constant of the eigenvalues (right panel). The smallest eigenvalue mode decayed faster (in green) and the largest slower (in blue), with the slower eigenmode largely determining the evolution of the state at later stages (in purple). The evolution along the two dynamic modes could be obtained by projecting the state at each time step onto the orthonormal basis defined by the two right eigenvectors (left and middle panels). The non-normal system, however, had non-orthogonal eigenvectors. In particular, we purposely set the eigenvectors to be closely aligned to each other, to obtain non-normal dynamics. This meant that the system state could no longer be decomposed using an orthogonal projection. Instead, the state was constructed from a linear combination of a non-orthogonal eigenvector basis (left and middle panels, were the parallelogram vector composition rule is applied for this). We provided the same input perturbation as for the normal system. Despite the similarities in design (in particular, having the same eigenvalues), the impulse response properties of this system were very different. Most notably, right after the pulse, there was a transient increase in the state norm (right and middle panels). However, the state eventually decayed to zero. This is because the long-term behaviour of a non-normal system is still governed by its eigenvalues. The smallest eigenvalue mode decayed faster and the largest slower, and they did so exponentially, as in the normal system. Yet, initially a transient amplification effect was observed. This happens precisely because of the difference in the decay rates of the modes, combined with the fact that the state is reconstructed with a non-orthogonal eigenvector basis (left and middle panels, note how the state vector, in purple, is reconstructed at each time step). The more aligned the eigenvectors are, and the stronger the difference between their eigenvalues, the larger the degree of non-normal amplification. Note that because of the non-trivial decomposition of the state along the non-orthogonal basis, the modes’ initial norm was very large (left and right panel, specially for the mode in blue). This resulted in the state experiencing a more sustained decay than in the normal system (right panel, compare blue and purple lines in both systems), i.e. in the input pulse to be transiently “persistent”.

#### 1.3 Alternative input modulation mechanism: fixed input subspaces with context-dependent external input signals

The *A, B ^{cx}* mechanism can be implemented even when the input dimensions are fixed across contexts

*B*. This is because the direction of the input vectors within the input subspaces

*B*

**u**can still be changed, provided that the external inputs are context-dependent

**u**

^{cx}, and that the dimensionality of the input subspaces is higher than 2D. This can be achieved by changing the external inputs’ strength in each context along each of the input dimensions (i.e. for 2D inputs, and ). This effectively changes the input vector

*B*

**u**coordinates within the orthonormal input basis

*B*. This change of coordinates can be used to rotate and stretch the inputs within the input subspace differently across contexts (i.e. and can be set to define any vector in the 2D input plane). This contextual mechanism might be closer to biology, given that long-range input projections are known to be anatomically stable.

### 2 Supplementary Figures

### 3 Supplementary Tables

## Acknowledgements

We thank Renate Krause for providing the RNN data and for valuable discussions. We thank Lea Duncker, Asma Motiwala, Saray Soldado Magraner and Gabriela Michel for providing feedback on the manuscript and for valuable discussions. This work was supported by the Gatsby Charitable Foundation.

## Footnotes

↵† Denotes shared senior authorship.