ABSTRACT
Behavior relies on continuous influx of sensory information about the body and the environment. In primates, cortex integrates somatic feedback to accurately reach and manipulate objects. Yet, in many experimental regimes motor cortex seems paradoxically to operate as a feedforward, rather than feedback-driven, system. Here, we recorded simultaneously from motor and somatosensory cortex as monkeys performed a naturalistic reaching and object interaction behavior. We studied how unexpected feedback from behavioral errors influences cortical dynamics. Motor cortex generally exhibited robust feedforward dynamics, yet displayed feedback-driven dynamics surrounding correction of behavioral errors. We then decomposed motor cortical activity into orthogonal subspaces capturing communication with somatosensory cortex or behavior-generating dynamics. During error correction, the communication subspace became feedback-driven, while the behavioral subspace maintained feedforward dynamics. We therefore demonstrate that cortical activity is compartmentalized within distinct subspaces that shape the population dynamics, enabling flexible integration of salient inputs with ongoing activity for robust behavior.
INTRODUCTION
Dexterous control of the large number of degrees of freedom in the arm and hand is a daunting task, further complicated b transmission delays that make the immediate effect of a motor command uncertain1. The brain appears to have solved this proble by predicting the effect of its commands, and updating future output based on the sensory feedback it receives Consequently, somatic feedback from the body is critical for the execution of accurate and robust movements: patients sufferin from the loss of cutaneous touch sensations or proprioception have great difficulty performing motor skills2. Understandin how the motor system incorporates somatosensory feedback is critical to understand how the brain generates behavior1. Th motor cortex is an appealing hub for such integration. Motor cortical neurons are the primary output of the cortex to th spinal cord3, but also broadcast commands to many parts of the brain including sensory regions like somatosensory cortex1 Motor cortical neurons respond rapidly to limb perturbations and other sensory inputs5,6, though it remains unclear ho sensory feedback is integrated within motor cortex throughout the generation of natural movements.
Despite the large numbers of neurons in motor cortex, the activity of individual motor cortical neurons largely reflects som underlying, lower-dimensional, population-wide dynamics7–12. The dynamics are captured within the neural manifold8, whic can be derived using dimensionality reduction techniques13. Recent work has shown that a large part of motor cortical activi within the manifold is well-modeled as a dynamical system14,15 wherein the future state of the neural population activity evolv deterministically from the present state. From these studies, it may seem that motor cortical activity can be viewed as feedforward generator of movement (i.e. a system that is not strongly influenced by external feedback inputs) with dynami that are inherently autonomous or self-driven. However, motor cortex requires constant input during the generation behavior16 and its activity is shaped by somatosensory feedback1. Why, then, do the motor cortical dynamics appear to b feedforward and not driven by feedback inputs?
We propose that even a largely feedback-driven system may appear to be feedforward if all inputs are predictable from the curre state of the population activity, such as during planned, practiced movements without errors or perturbations (Fig. 1a). Throughout these movements, the feedback inputs largely match what is expected from the outgoing motor command making it challenging to dissociate possible feedforward or feedback-driven components of motor cortical activit The arrival of unexpected inputs – such as somatic feedback from behavioral errors that deviate from the intended motor output – should lead to more apparently feedback-driven dynamics in the same cortical population (Fig. 1b). The degree to which a dynamical system is behaving as feedforward or feedback-driven can be quantified using trajectory tangling17, which occurs when identical neural states lead to diverging future states. A completely feedforward or autonomous dynamical system will have low tangling since the current activity deterministically predicts future activity, while high tangling indicates a system that is driven by unexpected inputs or is otherwise unstable. Prior analysis of motor cortical dynamics have used simple behavioral tasks with repetitive, well-practiced movements and found largely untangled, feedforward dynamics17. Such feedforward dynamics are advantageous as they allow cortex to produce a highly robust control signal that is resistant to noise17, including the biological limitations of neural circuits18 such as neuronal death19 or unreliable synapses20. If motor cortex is indeed input-driven16, more complex feedback-driven dynamics may only become apparent when studying naturalistic behaviors that include unexpected behavioral outcomes.
Can the motor cortex integrate relevant inputs while maintaining feedforward dynamics to robustly generate behavior, or is the population of neurons forced to switch between feedback-driven and feedforward regimes? Recent work has demonstrated that the neural manifold can be decomposed into subspaces capturing specific population-wide features. Subspaces can enable planning a movement7,11, adjusting motor output during learning21,22, sharing the control of a limb across hemispheres23,24, and even communicating between cortical regions25. We hypothesized that subspaces of neural population activity also enable effective coexistence of feedforward and feedback-driven dynamics (Fig. 1c). This framework allows cortex to operate as a set of linked, but independently-driven, systems comprising both feedback-driven and feedforward dynamics. Together, these systems shape the global cortical dynamics observed in the full population.
To test our hypothesis, we simultaneously recorded the activity of neural populations from motor and somatosensory cortex of four monkeys as they reached for and interacted with objects26. Using trajectory tangling17, we showed that the neural population dynamics substantially differed between the two cortical regions. Despite the complexity of the behavior, motor cortex exhibited untangled dynamics suggestive of a robust, feedforward system. In contrast, somatosensory cortex was comparatively more feedback-driven (tangled). As predicted, motor cortical dynamics became more feedback-driven (higher tangling) during behavioral errors and the subsequent correction.
Using a novel subspace decomposition algorithm, we showed that the motor cortex interacts with somatosensory cortex through a low-dimensional communication subspace, while an orthogonal low-dimensional behavioral subspace correlated with the behavior and exhibited feedforward dynamics. Intriguingly, the increased tangling in motor cortex during error correction was isolated only to its communication subspace, while the tangling of the behavioral subspace remained unchanged. Our work reimagines cortical activity as a composition of multiple function-specific systems isolated to distinct subspaces, and provides a novel framework for flexible integration of salient inputs with ongoing population activity.
RESULTS
A naturalistic reach and grasp task to study sensorimotor integration
The surprisingly feedforward nature of motor cortical dynamics observed in prior studies may be attributed to examining behavioral tasks performed by highly trained subjects that were allowed to fully plan movements before executing them10,14,15. To fully understand the effect of sensory feedback on motor cortical dynamics, we designed a more naturalistic task that resulted with occasional behavioral errors. In our task, monkeys reached for, grasped, and pulled objects mounted on a robotic arm26 (Fig. 2a). During the pulling phase, the robot acted as a spring, applying a resistive force proportional to the displacement. This task had several unique properties: 1) the task began with a ballistic reaching phase, similar to many prior studies11,14,27; 2) grasping the object provides a salient sensory cue; and 3) the resistive pulling phase required interactions with an external object. Four monkeys (macaca fascicularis) effectively learned to complete this task.
Due to the unconstrained nature of the task, each monkey exhibited an idiosyncratic behavioral strategy (Fig. 2c, left; Fig. S1). However, all monkeys occasionally made behavioral errors (Fig. 2c, pie charts; 18%, 29%, 27%, and 18% of trials for Mk-Cs, Mk-Br, Mk-Ol, and Mk-Jo, respectively) before ultimately succeeding in pulling the object. For Mk-Br, most errors occurred after grasping the object in order to adjust the posture of the hand to enable the pulling movement. For the remaining monkeys, errors were typically due to the hand missing the object or slipping off the object as it was pulled. These monkeys compensated by adjusting the position of the hand and grasping and pulling again, as shown by the second, smaller positive peak in hand velocity (Fig. 2c, right). These error trials provided an opportunity to study the effect of new, unexpected inputs on cortical activity.
As the monkeys performed the task, we simultaneously recorded kinematics of the left arm (Vicon, Oxford, UK), three-dimensional pulling force on the object, and the spiking activity of motor and somatosensory cortical neural populations (Fig. 2b) using chronically-implanted electrode arrays (Blackrock Microsystems, Salt Lake City, UT, USA). We isolated 78, 126, 62, and 88 motor and 82, 59, 44, and 42 somatosensory cortical neurons from Mk-Cs, Mk-Br, Mk-Ol, and Mk-Jo, respectively. Throughout the task, we observed striking similarities in the firing patterns of the somatosensory and motor cortical populations during the normal trials (Fig. 2d). On the error trials, we saw a transient reorganization of activity in both motor and somatosensory cortex around the time of error correction. For example, Mk-Cs (Fig. 2d, top row) had a prominent burst in activity in both regions around the time of error correction and subsequent pulling phase. The new activity patterns likely result from unexpected inputs arriving to both motor and somatosensory cortex reflecting the occurrence of the error, as well as altered activity driving the behavioral correction.
Motor and somatosensory cortex exhibit distinct dynamics during reach to grasp
To understand the change in neural responses observed on the error trials, we first characterized the dynamics of the somatosensory and motor cortical population activity. We applied Principal Component Analysis (PCA) to identify 20-dimensional manifolds8 that captured between 68-86% of neural population variance in each cortical region. We also applied PCA on the recorded arm kinematics and pulling force (behavioral covariates) to identify a 20-dimensional “behavioral manifold” capturing at least 95% of the variance. To quantify the local dynamical regime of each manifold, we used trajectory tangling17. High tangling values occur when the current state (e.g. a 20-dimensional vector in a neural or a behavioral manifold) leads to an array of vastly different future states. This is often true when the population is driven by unexpected inputs that change the temporal evolution of the state independently of its history. Low tangling values, in contrast, imply that the current state of the system accurately predicts future states. Importantly, even a feedback-driven system may appear to be feedforward if the inputs occur as expected in a manner that is consistent across trials. Since somatosensory cortex is driven by inputs from receptors in the limb28, we expected that its neural population dynamics should exhibit higher tangling than motor cortex, reflecting unpredictable changes of the somatosensory feedback17.
For the trajectories in the behavioral manifold, tangling was consistently very high for all four monkeys (Fig. 3a; Fig. 3b, left). The trajectories typically exhibited a cross-over point at reach onset and grasp, where most of behavioral variables are similar (hand at rest). Although somatosensory cortical activity is largely driven by feedback about the behavioral output, the trajectories of the somatosensory manifold (Fig. 3a, purple; Fig. 3b, middle) were significantly less tangled than in the behavioral manifold. Notably, the cross-over point was absent - the somatosensory cortical manifold trajectories during the movements followed a single path without prominent bifurcations. Nonetheless, the trajectories made sharp turns at different points, thereby causing the paths to frequently diverge and converge resulting in increased tangling. Despite the apparent similarities in the firing patterns of somatosensory and motor cortical neurons (Fig. 2d; Fig. S1d-e), trajectories in the motor cortical manifold were circular and smooth (Fig. 3a, green; Fig. 3b, right) resulting in significantly lower tangling than in both behavioral and somatosensory cortical manifolds17.
We then analyzed the manifold trajectories on error trials. Previous studies have shown that some motor cortex neurons respond rapidly to mechanical perturbations of the limb, despite the apparently feedforward nature of its population activity on normal trials1. We thus predicted that tangling of the motor cortex manifold trajectory increases when sensory inputs indicate a need to alter the current behavior, such as following the behavioral errors. In our task, each error was followed by a behavioral correction in order to complete the trial (Fig. 2b). These corrections corresponded to loops or distortions in the motor cortical manifold trajectories (Fig. 3c). We saw that motor cortical manifold tangling increased transiently surrounding the error correction (Fig. 3c-e), indicating that the motor cortex may then be driven by unexpected inputs. We observed similar increases in tangling for the somatosensory cortical manifold trajectories following behavioral errors (Fig. S3). We hypothesized that communication between motor and somatosensory cortex could account for the increased tangling in motor cortical activity surrounding errors.
We designed a simulation to validate DOS in a dataset with known ground truth dynamics (see Methods). We determined that DOS isolates specific dynamics with high accuracy on simulated data (Fig. S4). We then applied DOS to our behavioral, motor cortical and somatosensory cortical manifolds to identify communication and behavioral subspaces in motor cortical activity (Fig. 4a and S5a). Using cross-validation across all normal trials, we computed the neural variance explained by the communication and behavioral subspaces for assumed dimensionalities between one and five (Fig. 4b). For all four monkeys, the communication subspace explained the majority of neural population variance at all dimensionalities. Since the cost function of DOS does not optimize to explain variance in the motor cortical population, only covariance with somatosensory cortex, cortico-cortical communication appears to be a dominant feature of motor cortical population activity during movement. This observation is particularly surprising given the strikingly different dynamics of the full motor and somatosensory cortical manifolds (Fig. 3a-b).
Increased tangling of the motor cortex during error correction is isolated to its communication subspace
We next analyzed the dynamics within the motor cortical subspaces using tangling. Since the tangling metric is sensitive to dimensionality of the subspaces, we assumed both subspaces to be five-dimensional to facilitate direct comparison between them. If motor cortex receives unexpected inputs from somatosensory cortex during error trials, we hypothesized that the increase in tangling observed in motor cortical activity is isolated to its communication subspace. To ensure our results were not biased, we projected the activity on error trials into the subspaces previously identified using only the normal trials (Fig. 4c, Fig. S5).
Despite the increase in tangling observed within the full motor cortical population on error trials (Fig. 3c-e), we observed no significant difference in tangling within the behavioral subspace (Fig. 4d). In contrast, trajectories within the communication subspace increased in tangling around the time of error correction (Fig. 4e), similar to the full motor (Fig. 3c-e) and somatosensory (Fig. S3) cortical population activity. This increase in tangling was not observed in the remaining dimensions of population activity that were not captured by either the communication or behavioral subspaces (the null subspace; see Methods), indicating that the effect of unexpected somatosensory inputs related to the behavioral errors is confined only to the communication subspace (Fig. S6). These results together demonstrate that the increased tangling in motor cortex on error trials, indicative of a feedback-driven regime, is mediated using a communication subspace shared with somatosensory cortex.
DISCUSSION
We showed that during a naturalistic movement task composed of reaching, grasping, and pulling a loaded object, somatosensory cortical population activity appeared to be predominantly feedback-driven (Fig. 3), reflecting the tangled nature of the behavioral output. Despite the complexity of the task, motor cortical population activity exhibited smooth and untangled dynamics, characteristic of a dynamical system that is behaving in a feedforward manner. However, when the monkeys made a behavioral error, the motor cortical tangling acutely increased as if it were transiently more feedback-driven (Fig. 3c-e). The dynamics of the neural population activity in somatosensory cortex was more tangled than the motor cortical activity in both normal and error trials (Fig. 3a, Fig. S2, Fig. S3). This result was surprising given the apparent similarity of single neuron activity in the two regions (Fig. 2d). To study the interactions between the two cortical regions, we developed DOS, a subspace decomposition algorithm that isolated dimensions capturing cortico-cortical communication from those capturing behavioral output. In agreement with our hypothesis, the increase of tangling in motor cortex was confined to the communication subspace linking motor cortical activity to somatosensory cortex (Fig. 4). The remaining orthogonal components of motor cortical population activity (both the behavioral subspace and the null subspace) showed no change in tangling during the error trials (Fig. 4d, Fig. S6). These observations demonstrate that motor cortical activity is composed of multiple, independently-driven systems that together comprise the large-scale dynamics of the region.
Subspaces enable inputs to co-exist with ongoing population activity
Although inputs, whether sensory5,29 or cognitive such as during changes of mind30,31, influence motor cortical activity, the mechanisms that allow these inputs to coexist with ongoing population activity driving behavior have remained unclear. Intriguingly, motor cortex is dominated by apparently feedforward dynamics during the execution of planned or repetitive movements14,17,18. Modeling provides strong evidence that feedforward dynamics enables motor cortical populations to robustly generate motor output despite biological limitations of the neurons, such as neuronal noise, unreliable synapses or spontaneous death of neurons18. Nonetheless, motor cortex has to respond to external inputs when those inputs require unplanned changes of behavior. Here, we show how this apparent contradiction is resolved during correction of behavioral errors. The arrival of new information is compartmentalized into lower-dimensional subspaces of the neural manifold. This result has been challenging to observe in prior experiments focusing on highly trained, fully planned movements. This principle can only be uncovered by studying population activity during more complex behaviors where the possible arrival of inputs can be dissociated from the ongoing movements, such as the error trials in our naturalistic reaching and grasping task.
We studied the interactions between two brain regions, the somatosensory cortex and the motor cortex. However, a region such as the motor cortex receives input from many other regions in the brain. Through subspaces, our framework could allow any number of different brain regions to independently drive motor cortex. An interesting open question is how the independently-driven subspaces are linked to allow flexible sensorimotor integration and generate the full, population-wide dynamics. One intriguing possibility is that the link between the subspaces is direct but transient. Another possibility is that the subspaces are perpetually independent, but communication between them is mediated using additional dimensions within the neural manifold32 or even through other brain areas. Future theoretical and experimental work is needed to elucidate possible mechanisms underlying this integration.
Tangling as a means to quantify differences in dynamical regimes
The error trials demonstrate how independently-driven subspaces can allow motor cortex to adjust behavior. We found that motor cortex exhibited feedback-driven dynamics (quantified by an increase in tangling) transiently surrounding the error correction (Fig. 3c-e). Trajectory tangling is a compact and data-driven metric to quantify the local dynamical behavior of the neural population, i.e. to measure where the current state of a system is on a scale between a fully feedback-driven and a fully feedforward. The tangling value, however, only offers a relative measure of the dynamics of the neural and behavioral signals. To ensure that the tangling values are interpretable, we performed all comparisons between trajectories that were matched for dimensionality (20 dimensions for the manifolds, and 5 dimensions for the subspaces).
Surprisingly, the increase in tangling in motor cortex was contained to the communication subspace (Fig. 4d-e; Fig. S6). This result cannot be trivially explained by our methods. First, we derived the subspaces using only normal trials, ensuring that there was no bias enforcing that only the communication subspace captures the feedback-driven dynamics on error trials. Second, the tangling measure was not considered when defining the subspaces, ensuring that tangling remains an unbiased metric quantifying the dynamics. Third, the increased tangling is not a trivial consequence of covarying motor cortex with a more tangled cortical region like somatosensory cortex. There was no apparent bias for the communication subspace to have higher tangling on normal trials, and the behavioral covariates used to define the behavioral subspace had the highest tangling overall. Note that although we defined the communication subspace using covariance between motor and somatosensory cortex, our current experiment cannot causally implicate somatosensory cortex in driving these new motor cortical dynamics. Indeed, we would expect similar results if both cortical regions were driven simultaneously by a third region such as the thalamus16. Modern experimental tools that facilitate precise, real-time manipulation of neural circuits could help in future work to causally demonstrate the role of subspaces for cortico-cortical communication.
Decomposing population activity into subspaces
Several recent studies have leveraged population recordings from multiple areas to identify subspaces capturing putative cortico-cortical communication11,21,25, while others have employed targeted analysis methods to identify specific subspaces that capture covariance across conditions33, task epochs7, or with behavioral output34. The DOS algorithm expands upon many of these approaches by providing a flexible framework that jointly identifies multiple subspaces within the neural manifold capturing specific patterns of covariance with external signals. Here, we apply the algorithm to identify two subspaces within the motor cortical population activity, though it can be readily scaled to larger numbers of brain regions, as well as behavioral or environmental signals.
Activity of many cortical regions can be accurately represented by low-dimensional dynamics, despite the large number of neurons involved in their function8,21,35–38. Prior work has shown that untangled population activity allows for robust control signals within motor cortex17, but is not necessarily an ubiquitous phenomenon across all cortical regions. Here, we have shown in four monkeys that sensory cortical areas are more tangled than motor areas, as has been previously reported17. Yet, our subspace analysis shows that even within a single cortical region the neural population can be decomposed into independently-driven subspaces operating within distinct dynamical regimes that together shape the full population dynamics. In our experiments studying the control of reaching and grasping, motor cortical populations can leverage these subspaces to smoothly integrate new inputs. Broadly speaking, the high possible dimensionality of cortical populations enables a large number of inputs to be simultaneously integrated with ongoing computations by compartmentalizing them into distinct subspaces. Further experiments that simultaneously record from several interconnected cortical regions are needed to confirm whether cortex utilizes this approach to integrate multiple inter-regional inputs.
AUTHOR CONTRIBUTIONS
Conception and theory: M.G.P.
Experimental Design: M.G.P., S.C., M.B., A.B., B.B., J.B., G.C., S.M., M.C., T.M.
Surgical implantation: M.G.P., A.B., J.B., G.C.
Animal care: M.G.P., S.C., M.B., A.B., S.W., M.C.
Data collection and processing: M.G.P., B.B., S.C., M.B., A.B., S.W., M.C.
Data analysis: M.G.P.
Figure preparation: M.G.P.
Manuscript preparation: M.G.P.
Manuscript and figure revisions: M.G.P., M.C., T.M.
COMPETING INTERESTS STATEMENT
The authors declare no competing interests.
METHODS
Behavioral task
The monkeys were seated in a custom primate chair that enabled unconstrained interaction with a workspace in front of their body. The experimental platform and task has been previously described in detail26. In brief, a small, spherical, custom-molded, silicon object was affixed to the end of a seven-degree-of-freedom robotic arm (Intelligent Industrial Work Assistant, IIWA – KUKA, Augsburg, Germany). Custom control software enabled positioning of the robot in space. For Mk-Jo, only one position was provided, roughly centered in front of the body. For Mk-Cs, Mk-Br, and Mk-Ol, three positions were used: left, middle, and right. All positions were in the same coronal plane. The monkeys were trained to freely reach for the object, grasp it using only their left hand, and then pull it towards the body. In each trial, the robot brought the object to one of three randomly-selected positions (except Mk-Jo, where only one position was used). While the object was displaced from its starting position, the robot provided a force towards the starting position that was proportional to the horizontal displacement. Therefore, the monkeys had to resist this force to pull the object and, if they released it, the object quickly returned to its starting position. The system recorded the interaction forces produced by the robot.
Each trial was deemed successful after the position of the object passed a pre-determined distance threshold. Pulling movements that failed to reach this threshold were excluded. Upon success, Mk-Cs and Mk-Jo manually received a food reward, while Mk-Br and Mk-Ol received an automated liquid reward through a sipper tube. Error trials were manually identified by assessing the videos and kinematic trajectories. Most of the errors occurred when the monkey missed the robot in the initial reach or when their hand slipped off the robot while pulling. Typically, this resulted in errors preceding the final successful grasp, though for Mk- Br most errors occurred after grasping the object in order to adjust the posture of the hand to enable the pulling movement. While performing the task, the monkeys were affixed with up to 15 reflective markers to track kinematics of the joints of the arm and hand (Vicon Motion Systems, Oxford, UK).
Surgical procedures
All surgical procedures were performed using aseptic technique under general anaesthesia (induction: 0.1 mg/kg midazolam and 10 mg/kg ketamine; maintenance: 5 ml/kg/h propofol and 0.2-1.7 ml/kg/h fentanyl IV). A certified neurosurgeon (Dr. Jocelyne Bloch, CHUV, Lausanne, Switzerland) supervised all procedures. Each monkey was implanted with two Utah electrode arrays (Blackrock Microsystems, Salt Lake City, UT, USA). The first array targeted the arm region of motor cortex near the central sulcus (primary motor cortex; 1.5 mm shaft length) and the other targeted somatosensory cortex (Brodmann’s Area 2; 1 mm shaft length). The two arrays were connected to a single pedestal mounted to an implanted titanium that covered a portion of the skull. The mesh was molded to the shape of the skull of each monkey using a plastic skull 3-D printed from an individualized MRI scan. The implants in Mk-Cs and Mk-Br targeted arm areas identified by intraoperative surface stimulation (biphasic pulses, 3mA), while the implants in Mk-Jo targeted the hand areas. All arrays had 64 channels in an 8×8 configuration except the somatosensory cortical array of Mk-Br, which was 32 channels in an 8×4 configuration. Mk-Br and Mk-Ol each additionally received a third array (32-channel, 1mm shaft) anterior to the first array near to ventral premotor cortex. We saw qualitatively similar dynamics in this and the array implanted near the Central Sulcus. We therefore combined the signals recorded from both arrays in each monkey and for simplicity refer to this combined signal as “motor cortex”. In the same surgery, Mk-Br also received bipolar EMG electrodes in the left arm to record the activation of eight muscles of the arm and hand. Detailed surgical and post-operative care procedures have been described previously39.
Data acquisition
As the monkeys performed the behavioral task, we recorded infrared video from 12 cameras to track the kinematic markers at 100 Hz (Vicon Motion Systems, Oxford, UK) and neural activity at 30kHz using a Cerebus system (Blackrock Microsystems, Salt Lake City, UT, USA). To process the neural data, we bandpass-filtered each channel at 750 to 5000 Hz and set a threshold between −5x and −6.25x the RMS value to extract spike events. We sorted these spikes using Offline Sorter (Plexon, Dallas, TX, USA) to identify putative single neurons. We counted the number of spikes occurring in 10ms bins matched to the 100 Hz kinematic data. We used these binned spike counts in the Generalized Linear Model analysis (see below). For all other analyses, we square-root transformed each spike train to stabilize the variance13 and then converted them to an instantaneous firing rate by convolution with a Gaussian kernel with a 75 ms standard deviation.
Calculation of neural and behavioral trajectories
We applied a soft normalization procedure14 to all of the smoothed single neuron firing rates. We then used Principal Components Analysis (PCA) to reduce the dimensionality of each brain region independently and compute a neural manifold8. We selected the first 20 dimensions, though in practice our results were not impacted by the exact choice of dimensionality within reasonable ranges (10-30 dimensions; data not shown). We then computed a “behavioral manifold” using a similar procedure. First, we z-scored each of the position, velocity, acceleration, and force signals for the limb and robot. For Mk-Br, we also added the EMG recordings to this behavioral manifold to most accurately estimate the behavioral dynamics. We then applied PCA to this set of signals and selected the first 20 dimensions to correspond to the 20 dimensions of the neural manifold. The match in dimensionality was necessary for the subspace decomposition analysis described below, as well as to compare the trajectory tangling between manifolds.
Computing trajectory tangling
We employed trajectory tangling to quantify the relative dynamical behavior of the neural and behavioral dynamics. Tangling values should be highest at points in time where the current state could lead to multiple future states. We computed the instantaneous tangling (T) at each time t of the neural and behavioral trajectories using the formulation of 17.
The metric looks over all time points for all trials for each position, and is maximal when a similar position in state space (x) – which gives a small denominator value – corresponds to different state space velocities – which gives a large value in the numerator. The value of ε was chosen to be very small (10−6) and served only to keep the value from becoming undefined if the denominator was otherwise equal to zero. Note that we computed the tangling independently for all trials for each position, though we obtained similar results computing the tangling across all trials together. The formulation above, taken from Russo et al., was applied to trial-averaged data. Thus, noise and outliers were very unlikely. We modified the approach to take the 99.99th percentile across all time points in each position instead of the max in order to ignore a very small number of outliers (e.g. artifacts) in our single trial data.
Tangling values are sensitive to the dimensionality of the state space and, therefore, can be directly compared only between spaces with matched dimensionality. To facilitate easy comparison across signals or conditions, we ensured that dimensionalities were matched (20 dimensions for manifolds, 5 dimensions for subspaces). To study the time course of tangling changes, we interpolated to time-warp each trial to be the same length, preserving the proportion of time spent on average in the reaching and pulling phases. We then averaged across normal or error trials (e.g., Fig. 3d) To compute an aggregate single-trial tangling value, we computed the root-mean-square tangling through the trial (e.g., Fig. 3b) or throughout the specific period of behavioral error correction (e.g., Fig. 3e).
Decomposition into Optimal Subspaces (DOS) algorithm
We based DOS on a published method that jointly optimizes subspaces separating preparatory and movement activity within motor cortex7. We designed DOS to identify two orthogonal subspaces within the neural manifold, one capturing activity that covaried with the neural activity of another brain region (communication subspace) and the other capturing activity that covaried with behavior (behavioral subspace). Our method looks for eigenvectors spanning subspaces of the neural manifold that maximize the covariance with external covariates. The relevant value to be optimized is defined as:
Where:
Q: basis vectors of the subspace
C: covariance of neural activity and external covariates
d: dimensionality of subspace
σi: variance explained in the neural population by each subspace dimension
The numerator effectively measures the amount of covariance between neural activity and external covariates captured by the subspace. We defined C as the covariance between a matrix of N-dimensional covariates and N- dimensional neural population activity, which we can express as YTX. This formulation ignores a global scaling term for covariance since we are deriving unit basis vectors.
Using this metric, a cost function can be defined that optimizes for any arbitrary number of subspaces, with any value of d for each subspace, so long as the sum of their dimensionality is less than or equal to N. In our analysis, we defined two subspaces. For the behavioral subspace, Y was the activity within the 20-dimensional behavioral manifold identified from the behavioral covariates described above. For the communication subspace, Y was the activity within the somatosensory cortical manifold. These were referenced against X, the activity within the motor cortical manifold.
We defined the cost function using the above metric:
Subject to the constraints:
Where:
Qcom: basis of communication subspace
Qbeh: basis of behavioral subspace
X: local neural population activity (e.g. motor cortex)
Yneur: neural population activity of other brain region (e.g. somatosensory cortex)
Ykin: kinematic and behavioral covariates
dcom: dimensionality of the communication subspace
dbeh: dimensionality of the behavioral subspace
: covariance of local neural population activity with other brain region
: covariance of local neural population activity with behavioral covariates
The first two terms of the cost function maximize the ability of the communication subspace to explain the covariance of the neural population with the other brain region, as well as the ability of the behavioral subspace to explain the covariance of the neural population with the kinematic signals. The last term penalizes sparsity in the subspace weights. This term is maximized when the weight terms are evenly distributed across the dimensions. In practice, our results on the neural datasets were qualitatively similar without this sparsity constraint, however from our simulation experiments described below, this term was important to avoid edge cases. For example, if a high-variance PC can explain a lot of behavioral covariance, the algorithm may settle into local minima by attributing a high weight to one dimension. Instead, we want to bias towards finding distributed representations across dimensions. Typically, such sparsity constraints are achieved using the L1-norm. However, since the algorithm optimizes for a basis set, the L1-norm of Q is always 1 regardless of the weights. To capture the same intuition of penalizing sparse weightings to a basis set, we opted to use the root-mean-square of the weights. The α term helps to balance the contribution of the different terms.
The cost function used here can be viewed as analogous to that of Reduced Rank Regression, which specifically finds dimensions within the full population space that covary with other signals25. It is important to note that there is no constraint in the cost function to explain neural variance, only to explain the covariance with external signals. In theory, one of the subspaces could actually explain very little neural variance. The optimization was performed using a Matlab-based manifold optimization framework40. Thus, our algorithm can jointly optimize multiple orthogonal subspaces within the same neural population, each subspace designed to explain the covariation of the full population with external signals. A key assumption in the above algorithm is that the dimensionality is matched across all sets of signals, i.e., that Cbeh and Ccom are square and of identical size. We achieved this by performing PCA on the population activity and kinematic signals before running this algorithm and selecting the same number of dimensions for all sets of signals. We used DOS to identify motor cortical communication and behavioral subspaces from neural activity and behavior covariates from normal trials only. To analyze the error trials, we projected the neural activity from the error trials onto these subspaces identified from normal trials. This ensured that the error analysis results were not biased by the optimization algorithm.
We quantified the amount of variance captured by each of the behavioral and communication subspaces using the eigenvalues of the covariance matrix between the subspaces and the activity in the motor cortical manifold.
Simulation to validate the DOS algorithm
We simulated population of spiking motor and somatosensory cortical neurons where the ground truth latent dynamics are known to confirm that the above decomposition algorithm can correctly reconstruct the latent dynamics in the intended subspaces. We generated two sets of latent dynamics from a one-dimensional random walk, termed behavioral and communication. We then smoothed these random dynamics with a Gaussian kernel of 50 ms width to obtain smoothness that is comparable to that of the neural data. We developed DOS with intention to identify specific patterns of covariance, even in the face of confounding correlated signals. To test the accuracy of the algorithm, we added two confounds. First, both the motor and somatosensory cortical populations received correlated sinusoidal dynamics to simulate the effect of common input to both regions (referred to as null dynamics). Second, the somatosensory cortical population received the precise communication dynamics as well as a time-shifted version of the behavioral dynamics meant to simulate the real-world delays between intended motor output (here assumed to be 50ms) and the corresponding sensory feedback (assumed to be 50ms).
The motor and somatosensory cortical population activity was generated as a Poisson process whose input was randomly weighted combinations of the null, communication, and behavioral dynamics. We then processed the simulated spiking activity with the same methods as the real neural data, including normalization, smoothing, and dimensionality-reduction through PCA. We identified the subspaces within the simulated motor cortical data using the covariance between motor and somatosensory cortical PCs, as well as the covariance between motor cortex and the known behavioral dynamics. We quantified the performance by computing the variance explained (R2) between the ground truth dynamics and the extracted subspace dynamics. We repeated this random simulation 1000 times to generate the performance distributions. On each simulation, we compared the performance of DOS to a null distribution found by taking the max correlation across all individual motor cortical PCs. This null distribution served to confirm that the reconstructed dynamics did not trivially arise from covariance across neurons found by PCA, but instead required a targeted approach like DOS.
ACKNOWLEDGEMENTS
We thank Maude Delacombaz and Melanie Kaeser for assistance with behavioral training of the animals. M.G.P. thanks Dr. Juan A. Gallego and Dr. Raeed H. Chowdhury for their valuable discussions in the preparation of this manuscript. Funding: Whitaker International Scholars Program fellowship (M.G.P.), Wyss Center for Bio and Neuroengineering (WCP-008, G.C., M.C. and T.M), GTX medical (G.C, M.C), Bertarelli Foundation Catalyst Fund (BC1709, T.M. and M.C.), the Bertarelli Foundation, and the Swiss National Science Foundation through Ambizione awards (PZ00P2_168103, T.M.; PZ00P2_167912, M.C.), the NeuGrasp project (170032, S.M.), and the Swiss National Centre of Competence in Research (NCCR) Robotics.
Footnotes
↵* these authors jointly supervised this work