Skip to main content
bioRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search
New Results

Stimulus-choice (mis)alignment in primate MT cortex

View ORCID ProfileYuan Zhao, View ORCID ProfileJacob L. Yates, Aaron J. Levi, Alexander C. Huk, Il Memming Park
doi: https://doi.org/10.1101/2019.12.20.884189
Yuan Zhao
Department of Neurobiology and Behavior, Stony Brook University, Stony Brook, New York 11794, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Yuan Zhao
Jacob L. Yates
Brain and Cognitive Science, University of Rochester, Rochester, NY, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Jacob L. Yates
Aaron J. Levi
Center for Perceptual Systems, Departments of Neuroscience & Psychology, The University of Texas at Austin, Austin, Texas 78712, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Alexander C. Huk
Center for Perceptual Systems, Departments of Neuroscience & Psychology, The University of Texas at Austin, Austin, Texas 78712, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Il Memming Park
Department of Neurobiology and Behavior, Stony Brook University, Stony Brook, New York 11794, USA
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: memming.park@stonybrook.edu
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Preview PDF
Loading

Abstract

For stimuli near perceptual threshold, the trial-by-trial activity of single neurons in many sensory areas is correlated with the animal’s perceptual report. This phenomenon has often been attributed to feedforward readout of the neural activity by the downstream decision-making circuits. The interpretation of choice-correlated activity is quite ambiguous, but its meaning can be better understood in the light of population-wide correlations among sensory neurons. Using a statistical nonlinear dimensionality reduction technique on single-trial ensemble recordings from the middle temporal area during perceptual-decision-making, we extracted low-dimensional neural tra jectories that captured the population-wide fluctuations. We dissected the particular contributions of sensory-driven versus choice-correlated activity in the low-dimensional population code. We found that the neural trajectories strongly encoded the direction of the stimulus in single dimension with a temporal signature similar to that of single MT neurons. If the downstream circuit were optimally utilizing this information, choice-correlated signals should be aligned with this stimulus encoding dimension. Surprisingly, we found that a large component of the choice information resides in the subspace orthogonal to the stimulus representation inconsistent with the optimal readout view. This misaligned choice information allows the feedforward sensory information to coexist with the decision-making process. The time course of these signals suggest that this misaligned contribution likely is feedback from the downstream areas. We hypothesize that this non-corrupting choice-correlated feedback might be related to learning or reinforcing sensory-motor relations in the sensory population.

Author summary In sensorimotor decision-making, internal representation of sensory stimuli is utilized for the generation of appropriate behavior for the context. Therefore, the correlation between variability in sensory neurons and perceptual decisions is sometimes explained by a causal, feedforward role of sensory noise in behavior. However, this correlation could also originate via feedback from decision-making mechanisms downstream of the sensory representation. This cannot be resolved by analyzing single unit responses, but requires a population level analysis. Area MT contains both sensory and choice information and is known to be the key sensory area for visual motion perception. Thus the decision-making process may be corrupting the sensory representation. However, we find that the sensory stimuli and choice variables are separate at the population level,contradicting the previous interpretations based on single unit recordings. This new insight postulates how neural systems can maintain a mixed representation while allows learning and adaptation.

Introduction

Sensory cortical neurons exhibit substantial variability to repeated presentations of the same stimulus [1, 2]. This variability depends on the specifics of the sensory stimulus and task being performed [3–7], and is often correlated with the trial-by-trial perceptual report of the animal [8–11]. This trial-by-trial correlation between neural responses and perceptual reports, often quantified as choice probability (CP), has long been of interest for its potential to reveal the mechanisms by which downstream areas read out the response of relevant population of sensory neurons [12–14]. However, this interpretation is complicated by the presence of interneuronal correlations [15], top-down feedback [9, 16] and also depends on assumptions about the readout mechanisms of downstream brain areas [12, 14, 16, 17].

Several models of perceptual decision-making have been proposed to explain the empirical relationships between stimuli, neural responses, and behavioral choices [12, 14, 16]. Existing proposals come in two basic flavors: those that posit an optimal readout that is limited by shared neural variability [14, 18, 19] and those that assert that choice-related feedback modifies the signals in sensory areas [16, 20]. Several recent experimental results support the feedback hypothesis [7, 9, 20]. Although feedback can be interpreted in terms of probabilistic inference [16], the resulting pattern of variability in sensory areas will reduce the information about the stimulus [16, 19, 21] and impair performance on the task [20]. Why would the brain bother to feedback a choice or decision that corrupts the sensory information and make it do worse on the task? Here, we propose an alternative hypothesis: that the feedback can be non-corrupting, effectively multiplexing choice signals in a sensory population without diminishing information about the stimulus.

To visualize the space of hypotheses and how they can be distinguished, it is helpful to summarize the joint activity of a population of neurons with respect to the stimulus driven activity. Figure 1 demonstrates this alignment conceptually and the effect of each type of choice model in this space. Specifically, for a population of only two neurons, the joint activity of the population can be represented as points in a 2D space where each axis represents an individual neuron’s activity (Figure 1A). For a one-dimensional stimulus (as is typically used in discrimination paradigms), different values of the stimulus (red and black) will drive activity that falls along a one-dimensional “stimulus axis”. Increased variability along the stimulus axis will decrease the amount of information about changes in the stimulus, while, importantly, variability orthogonal to the stimulus axis will not [19, 22]. We call this variability the “non-stimulus subspace” (Figure 1A). In larger populations, it is possible that the “stimulus axis” could be higher than one-dimension, however, there will still be a subspace that is orthogonal to the stimulus axes and, therefore, will not affect information about the stimulus (i.e., in the null-space of the stimulus axes).

Fig 1.
  • Download figure
  • Open in new tab
Fig 1.

Hypotheses on the sources of choice correlations in sensory area. (A) Joint activity of the population. The point cloud represents neuronal activities colored by stimulus direction. The neural space can be divided into stimulus and non-stimulus spaces. (B) Noise correlation is any elongation of the joint activity point cloud for repeats of the same stimulus. (C) Optimal readout. The optimal decision boundary is a criterion line orthogonal to the stimulus axis. All CP is due to readout and there is no CP in the non-stimulus subspace. (D) Suboptimal readout. The decision boundary is not orthogonal to the stimulus axis. CP exist in both subspaces. (E) Corrupting feedback. The choice is fed back and pushes variability along the stimulus axis. This increases CP along the stimulus axis without affecting the non-stimulus subspace, and causes more variability along the stimulus axis. (F) Non-corrupting feedback. Feedback pushes choice information in the non-stimulus subspace and increases CP in the non-stimulus subspace without adding CP in the stimulus axis.

By realigning the population activity to the “stimulus axis”, the effect of noise correlations and feedback can be visualized clearly. Noise correlation is any elongation of the joint activity point cloud for repeats of the same stimulus (Figure 1B). In this space, the optimal readout of such a population is to draw a criterion line (decision boundary) orthogonal to the stimulus axis and report which side the population activity on that trial fell on (Figure 1C). The classic feedforward hypothesis, i.e. optimal readout limited by noise, produces CP greater than chance along the stimulus axis because noise correlations pushed the population activity across the two sides of the discrimination boundary. In this model, all CP is due to readout and there is no CP (i.e., 0.5) in the non-stimulus subspace. In corrupting feedback models [16, 20, 21], the choice is fed back only along the stimulus axis. This increases CP and causes more variability along the stimulus axis without affecting the non-stimulus subspace, and reduces the performance on the task for weak stimuli (Figure 1E). In contrast, feedback could be non-corrupting by pushing choice information only in the non-stimulus subspace (Figure 1F). This increases CP in the non-stimulus subspace without adding CP in the stimulus axis and does not diminish stimulus decoding performance. In each of these examples (Figure 1C,E,F) the readout is optimal (orthogonal to the stimulus axis). For completeness, one additional possibility is that the readout is suboptimal and the downstream areas are mistakenly including variability that is in the non-stimulus subspace giving rise to CP in the non-stimulus subspace (Figure 1D).

To test these different hypotheses requires an analysis of the joint statistics of populations of sensory neurons while subjects perform a discrimination task. Here, we take advantages of recent developments in statistical dimensionality reduction of single-trial population recordings [23] to examine how information about the stimulus and choice are encoded jointly in small populations of simultaneously recorded MT neurons during perceptual reports about integrated motion direction [24]. The effects of stimulus, choice, and trial-to-trial variability present in the population activity are decomposed into shared low-dimensional neural trajectories and noise that is private to each neuron (Figure 2). As expected, low-dimensional shared signals capture a majority of the variability in these data as seen previously in other areas [6, 23, 25, 26]. By aligning the latent signals to the stimulus and task variables, we were able to investigate how stimulus and choice are encoded by neurons collectively.

Fig 2.
  • Download figure
  • Open in new tab
Fig 2.

Probabilistic description of a single trial using variational latent Gaussian process method and resulting noise correlation. (A; top) Simultaneously recorded spike trains of the MT units in an example trial (yt in Eq. 1). (A; bottom) Corresponding 4-dimensional trajectory. The rank-4 matrix multiplication of the loading matrix (matrix A in Eq. (1)), and latent trajectory is exponentiated to produce the population rate. The loading matrix is rotated to maximize stimulus encoding (see Fig. 3), so that the first column has the strongest stimulus response. The inferred latent factors (xt in Eq. (1)) are colored to indicate the respective factors corresponding to the loading matrix. (B) The noise correlation matrices (neuron by neuron) for the sessions with frozen trials. The lower triangles are the correlations calculated from the raw data, and the upper triangles are the correlations from the reconstruction by the inferred 4-dimensional latent factors. Time bin size 100 ms.

We found that the task variable (visual motion) was primarily captured by a single latent dimension, indicating that the high-dimensional visual stimulus was represented in a low-dimensional, task-relevant manner across the MT population. Additionally, we found that the choice-correlated variability in the population was mainly captured by the latent subspace orthogonal to the task dimension. These results suggest the monkeys’ readout is either highly suboptimal or choice signal is fed back to sensory cortex in the null space of the stimulus – multiplexing choice signals in sensory areas without corrupting information about the stimulus. This feedback signal could be critical for adapting sensory representations while learning new tasks or in non-stationary environments [27, 28].

Materials and methods

Electrophysiology, task, and behavioral data

Data were recorded from three adult rhesus macaque monkeys (two males, P & L, and one female, N) performing perceptual decision-making task for multiple sessions (P: 9, L: 13, N: 10) as reported in [24, 29] (with additional sessions added). Spike trains from area MT were obtained via linear electrode arrays (Plexon U or V Probe). All procedures were performed in accordance with US National Institutes of Health guidelines, were approved by The University of Texas at Austin Institutional Animal Care and Use Committee. The stimulus consisted of a hexagonal grid of drifting or flickering Gabor patches. In each trial, the stimulus consisted of 7 consecutive motion pulses, each lasting 150 ms. The strength and direction of each pulse was randomly drawn from Gaussian distribution and rounded to integers. The monkey was rewarded for making a correct choice if the total sum of motion pulses was greater in the corresponding direction. The monkey was rewarded at random with probability 0.5 on the zero sum trials. We keep the recordings from 100 ms before the visual stimuli onset to 350 ms after the visual stimuli offset. We analyze sessions with at least 10 neurons in order to extract latent processes (for a total of 14 sessions). Length of sessions ranged from 245 to 1000 good trials.

Single-trial latent dynamics of population

To understand how stimulus and perceptual choice are encoded across the population, we employed the variational latent Gaussian process (vLGP) method to extract single-trial low-dimensional neural trajectories from population recordings in area MT. We used the recording of the period from 100 ms before stimulus onset to 350 ms after offset, and binned the spike counts at 1 ms resolutions. Let xk denote the k-th dimension of the latent process. We assumed that the spatial dimensions of latent process are independent and imposed a Gaussian Process (GP) prior to the temporal correlation of each dimension, Embedded Image

To obtain smoothness, we used the squared exponential covariance function and respective covariance matrix K in the case of discrete time. Let ytn denote the occurrence of a spike of the nth neuron at time t, ytn = 1 if there was a spike at time t and ytn = 0 otherwise at this time resolution. Then yt is the vector of length N, total number of neurons in a session, that concatenates all neurons at time t. The spikes yt are assumed to be a point-process generated by the latent state xt at that time via a linear-nonlinear model, Embedded Image

To infer the latent process (xt for each trial) and the model parameters (A and b), we used variational inference technique, as the pair of prior and likelihood do not have an tractable posterior. We assumed parametric variational posterior distribution of the latent process, Embedded Image

We analyze the mean {μk} as the latent trajectory in this study. The detail of inference is described in [23]. To accelerate the inference, we initialized algorithm at the result of Gaussian Process Factor Analysis (GPFA). The dimensionality of the latent process was determined to be 4 by leave-one-neuron-out cross-validation on the session with the largest population (2). All the sessions with more than 10 simultaneously recorded units were included in the this study.

Pulse-triggered average

To measure the relationship between the time-varying pulse strength and the inferred latent process, we measured the contribution of pulses to the latent process. The pulse-triggered average (PTA) measures the change in latent process resulting from an additional pulse at a particular time of unit strength. To calculate the PTA, we used the pulse stimulus and latent response at 1 ms resolution. For each session, let si denote the value of the i-th motion stimulus, and let xtk denote the k-th dimension of the latent process at time t. All trials were concatenated such that the latent process X is a matrix of length T × 4, where T is the total time. For the i-th pulse, si is the number of Gabors pulsing, with si > 0 for pulses in one direction and si < 0 for pulses in the other direction. To calculate the temporal lags of the PTA, we built design matrices, D = [D1,D2,…,D7]. For the i-th pulse, the design matrix Di is a T × 28 matrix that consists of 4 cosine basis functions at the 4i + 1, 4i + 2, …, 4i + 4-th columns and 0 elsewhere. These basis functions starts at 0 ms, 50 ms, 100 ms and 150 ms after the onset, lasts 100 ms each and spans the rows of Di. The magnitude of the bases is equal to the corresponding pulse value si. We calculated a separate Di for each of the seven pulses and concatenated them to obtain a design matrix for all seven pulses and estimated the weights with ℓ2-regularization, Embedded Image where W is the weight matrix to estimate and E is the Gaussian noise matrix and the regularization hyperparameter γ was chosen by the generalized cross-validation (GCV) [30]. The PTA was calculated with the design matrices of unit-strength pulse and the estimated weights β. We smoothed the PTA with a temporal Gaussian kernel (40 ms kernel width).

Subject to arbitrary rotations, a latent trajectory forms an equivalence class of which the members have the same explanatory power in the vLGP model. We seek a particular rotation for each session that makes the encoded task signal concentrate in the first few dimensions. By singular value decomposition, WT = USVT, we rotate the trajectory x to UTx.

Choice decoder

Since there were some recording sessions with less than ideal number of frozen trials for the calculation of choice probability, we instead analyzed the “weak” trials of which the monkeys’ correct rate was below a threshold (65%). We started at the trials of zero pulse coherence and gradually increased the magnitude of coherence (absolute value) until the correct rate reached the threshold. One of the sessions containing less than 100 weak trials was excluded in this analysis.

We removed the stimulus directions that are encoded in the latent process and raw population activity of weak trials by regressing out the pulses and analyzed the residuals. The latent process and population activity were re-binned at 100 ms resolution where the value of each bin is the sum of latent state xt or spike counts yt over the bin for t = 1, 2, …, T. For each t, we assumed a linear model to predict its value Embedded Image where si denote the strength of the i-th pulse, wti is the weight vector corresponding to the bin and pulse, and e is the homogeneous Gaussian noise across all bins. We estimated the weight vector by least-squares with ℓ2-regularization to prevent over-fitting, Embedded Image

Again, the hyperparameter of regularization was chosen by GCV. For the raw population activity, we did the same regression, replacing xt with the spike count yt. We then analyzed the contribution of behavioral choice on the residuals Embedded Image

For the whole trial we used the sum residual of the windows r = Σt rt. The range of t depends on the period of interest.

We trained logistic models, to which we refer to as choice decoders, to predict the subj ect’s choice on each trial using either latent trajectories or population responses. The weights β and bias β0 were estimated by maximum likelihood with ℓ2-regularization, Embedded Image

Due to small sample sizes, the hyperparameter of regularization was chosen via 3-fold stratified (balanced classes in test set) cross-validation for every session individually.

Choice mapping

The conventional choice probability only applies to univariate variables. However, either the latent process or population activity is multivariate. We transformed the multivariate variables mentioned above onto a one-dimensional subspace that has the same direction as the choice through the choice decoders, Embedded Image

We refer to the transform as the choice mapping. The quantity c is a normalized value within [0, 1] that maps the residual onto the choice direction [31], and enables aggregation across sessions as well. We pooled these mapped values of test sets and aggregate them across all sessions. By plugging different dimensions of latent process or population activity as r in the mapping, we obtained the choice-mapped values of the stimulus-dimension, non-stimulus-dimensions of latent process and the whole population. With these mapped values, we calculated the choice probability of the corresponding dimensions.

To investigate the effect of different dimensions on the choice, we did sequential likelihood ratio tests through adding the choice-mapped value of stimulus-dimension, non-stimulus-dimensions and the population one by one to a logistic model that predicts the choice, Embedded Image

To investigate the time course of choice probabilities, we used choice decoders to perform choice-mapping on the whole data with a 100 ms non-overlapping moving window. The choice decoders were fitted to early (200–500 ms), middle (600–900 ms) and late (1000–1300 ms) periods of non-stimulus latent dimensions(Figure 6). The decoders were regularized with cross-validation as mentioned before. The choice probabilities were then calculated based on the choice-mapping using the decoders.

Results

Low-dimensional shared variability structure

Three monkeys performed a motion-pulse direction discrimination task with an eye movement to one of two targets [29]. The visual stimulus was presented as a sequence of 7 temporally coherent motion pulses of varying strength. An ensemble of MT neurons were simultaneously recorded using multi-electrode arrays. Given the recording, we statistically infer a low-dimensional latent process that explains the shared component of the high-dimensional variations in the observed spiking activity. Conventional analysis methods such as factor analysis or principal component analysis assume either observation models inappropriate for spikes (e.g. Gaussian) or linear dynamics that lack expressive power to describe any non-trivial computation. To overcome these disadvantages, we imposed a general (nonlinear) Gaussian process prior on the latent trajectories and assumed a point-process observation model to account for spikes. The generative model was fit using the variation latent Gaussian process (vLGP) method to recover nonlinear smooth latent trajectories from population recordings [23]. Figure 2A shows the scheme of the model and an example trial. The population firing rates are driven by the latent trajectory through a linear-nonlinear cascade. The loading matrix linearly maps the high-dimensional observation space to the low-dimensional latent space, of which the rows corresponding to the neurons and the columns corresponding to the latent dimensions. The extracted latent trajectories captured the shared variability of the population activity, while the individual variability of each neuron was explained by stochastic generation of spike trains. The dimensionality of latent trajectory was chosen to be 4 by a leave-one-out cross-validation scheme on the session with the largest population (N = 21 neurons). To aggregate analysis across sessions, we fixed this dimensionality of the latent processes.

To validate the model, we evaluate the pairwise noise correlations between neurons on randomly interleaved frozen trials where the stimulus was held constant (Figure 2B). With the inferred latent process and loading matrix, we can generate spike trains from the model. We calculated the noise correlation matrices from data and reconstructed spikes (Figure 2B). The results show that the extracted latent trajectories captures well the co-variability of the population with only 4 dimensions.

Stimulus-encoding is concentrated in one shared dimension

In previous work, MT neurons showed strong transient responses on average to motion pulses [24]. We ask if the individual MT responses to visual stimulus are aligned at the population level. To describe the temporal dependence of the latent process on the motion pulses, we calculated the pulse-triggered average (PTA) for each of the seven pulses [24]. The PTA captures the change in latent state resulting from a unit visual motion (a single patch of Gabor drifting in one direction during a pulse), assuming a linear scaling with motion strength (see Materials and Methods).

The latent process is subject to arbitrary rotation [23] which results in models with equivalent explanatory power. Hence, we rotated the latent processes for each session so that the effects of motion pulses are concentrated in decreasing order across dimensions (Figure 3A). For both subjects, the pulses are faithfully represented as transiently modulated latent process, and most of the motion information is encoded in the mean value of the first factor—we refer to this factor as the stimulus axis.

Fig 3.
  • Download figure
  • Open in new tab
Fig 3.

Visual motion pulse information encoded in one dimension. (Pulse-triggered average. Three example trials from each monkey are shown. The factors are rotated such that most of the stimulus power is in the first factor. The color gradient indicates the seven pulses of visual motion stimuli. (bottom right) The power of each factor that explains the variation contributed by the stimuli to the factors. The shape indicates the animal and the color indicate the respective trials.

We pooled the stimulus-explaining latent factor alignment across all sessions. The first dimension explains most (> 90%) of the PTA in the latent process for all but one session (Figure 3B). This concentration of stimulus information in 1-dimension is consistent with the canonical view of MT as primarily a sensory area. Since the sensory stimulus is 1-dimensional (directional motion with different strength), this suggests that the encoding of MT units is temporarily uniform (without multiple time scales of adaptation or lag) and linear (no nonlinear superposition). Note that this is not a trivial result, since the motion information can be encoded in a curved 1-dimensional manifold that spans multiple dimensions in the neural space [32].

Sensory and choice population codes are misaligned in MT

Next, we investigate how the downstream choice signal is aligned with respect to the stimulus axis. There are several possibilities that the choice-correlations can manifest in the MT population activity (Figure 1). To optimally perform the task, the choice should rely only on the stimulus and ignore the off-axis “noise” [17]. Hence, for a purely feed-forward system, only the noise in the stimulus dimension should influence the choice, resulting in choice-correlation reflecting the optimal strategy (Figure 1C). Otherwise sub-optimal “readout” can show choice-correlation through stimulus-irrelevant variability (Figure 1D). On the other hand, feedback paths can mix the downstream choice process signals back into the MT representation: if the feedback is aligned with the stimulus-axis, it will corrupt the encoding of the sensory signal (Figure 1E), while misaligned feedback that stays orthogonal to the continuous stream of stimulus modulated population activity subspace (Figure 1F).

To investigate the effect of different axes on the choice, we calculated the choice probability of the recorded neural population after mapping the multivariate activity to choice through choice-mapping (Figure 5; see Materials and Methods). The pooled choice probability estimated using the choice-mapped stimulus-axis, non-stimulus-axes (the 3-dimensional subspace orthogonal to the stimulus-axis), and all 4 dimensions of the MT latent tra jectory are 0.546, 0.591, and 0.621 respectively (Fig. 4). The estimated population spike count choice probability is 0.627. To verify that the pooling across sessions (Fig. 5, stage 4) does not weaken the choice information, we compare the pooled model with models of individual sessions as a baseline. The likelihood ratios of full models of individual sessions to the full model of pooled sessions is between 0 and 1 because the log-likelihoods are always negative and the pooled model is at most as good as the individual session models. Among the 10 sessions, the likelihood ratio ranges between 0.92 and 0.99, and the average is 0.98. These indicate that the pooling over sessions keeps most of the choice information.

Fig 4.
  • Download figure
  • Open in new tab
Fig 4.

Choice probabilities of latent factors for each monkey. Contours corresponds to 50%, 90%, 99% quantities of the choice-mapped stimulus and non-stimulus trial distribution. The IN choice distribution (red-shade contours) is biased upward, indicating existence of the choice information in the non-stimulus axes. The pooled choice probability estimated using the choice-mapped stimulus-axis, non-stimulus-axes (the 3-dimensional subspace orthogonal to the stimulus-axis), and all 4 dimensions of the latent trajectory are 0.546, 0.591, and 0.621 respectively. The estimated population spike count choice probability is 0.627. For nested statistical tests of the corresponding regression models, see main text and Fig. 5.

Fig 5.
  • Download figure
  • Open in new tab
Fig 5.

Data analysis pipeline and nested model comparison. (1) Extract latent processes. (2) Align latent processes to stimulus & null dimensions. (3) Map dimensions of latent processes into real-valued scalars. (4) Pool the choice-mapping over all sessions and perform nested log-likelihood tests.

To determine the major contribution among regressors on choice, we performed nested likelihood ratio tests by adding the choice-mapped value of stimulus-axis, non-stimulus-axes, and the population, one by one (Figure 5). The choice is significantly correlated with the latent nonstimulus-axes (p < 2.2 × 10-16), which indicates that the choice axis is not perfectly aligned with the stimulus axis as the optimal readout or corrupting feedback models suggest. Therefore, our analysis supports representation of choice information in the non-stimulus latent subspace. This misalignment of stimulus axis and choice axis can occur through either non-optimal readout (Figure 1D) or non-corrupting feedback (Figure 1F). The misalignment between choice and stimulus in MT provides evidence for a feedback source of choice information in sensory neurons. The presence of CP orthogonal to the stimulus axis suggests that choice information is not just a result of noise on the sensory response, but rather arises from another process altogether.

Time course of choice probability indicates feedback of decision-making process to MT

The misalignment between choice and stimulus in MT suggests a feedback source for choice-correlated activity, but could still be explained by suboptimal readout. Debates based on models and arguments in the literature have yet to resolve issue of feedforward versus feedback choice correlations in area MT [16, 20, 33–35]. To disambiguate the two, we investigate the temporal profile of choice probability. Behavioral analysis showed that the sensory information immediately after its presentation has a strong influence in the choice [24]. In turn, one would expect to see choice information early in the population activity. If the choice information is only present late in the trial, then we can conclude that the feedback from the downstream decision-making process is contributing to the misaligned choice information we observed in the previous section.

To investigate the temporal profile of choice correlation in the non-stimulus axes, we calculated time course of CP. We fit 3 linear choice decoders to the latent non-stimulus axes during the early (200–500 ms), middle (600–900 ms) and late (1000–1300 ms) periods, and then used them to decode the whole period with a 100 ms moving window. Figure 6 shows that the middle and late decoders start climbing late during the visual motion presentation and reach a peak at around the motion stimulus was terminated. This temporal profile is consistent with a choice variable that accumulates sensory evidence [12], and supports the non-corrupting feedback from the decision-making process. On the other hand, the early decoder shows a constant choice probability throughout the motion presentation period (Fig. 6) which could represent a per-trial choice bias. These observations suggest that the choice information resides in more than 1-dimension within the non-stimulus subspace.

Fig 6.
  • Download figure
  • Open in new tab
Fig 6.

Time course of choice probability in the non-stimulus subspace suggests feedback from the decision-making process. Decoders were fit to early (yellow), middle (red), and late (purple) periods (300 ms, marked by the colored bars) of non-stimulus latent dimensions to predict choice. We used the resulting weights of the decoders to perform choice-mapping on the whole time interval divided into 100 ms non-overlapping moving windows (aligned at the center). The colored curves correspond to the choice probability time course using the respective decoder.

Conclusion

To understand how stimulus and perceptual choice are encoded across the population of MT neurons, we take advantage of recent developments in unsupervised statistical approaches to single-trial population analyses (Figure 7). Using vLGP, the stimulus, choice, and trial-to-trial variability presented in the population activity are decomposed to reveal the underlying signals: individual neuron’s private activity, and low-dimensional shared signals. As expected, latent low-dimensional shared signals capture the majority of the variability present in the population recordings. By aligning the latent signals to the stimulus and behavioral choice, we were able to investigate how stimulus and choice are shared across neurons. We found that the sensory task variable was primarily captured by a single latent dimension, indicating that high-dimensional visual stimulus was represented in a low-dimensional, task-relevant manner across the MT population. Surprisingly, we found that the choice-correlated variability in the population of was mainly captured in latent subspace orthogonal to the stimulus-encoding dimension, which suggests that either the downstream decision circuit used suboptimal readout from MT response (Figure 7, path 3) or the feedback from downstream circuit was non-corrupting (Figure 7, path 4). Further analysis of the time course of choice probability supports feedback mechanism rather than readout. Therefore, we propose a new multi-dimensional model of information representation within MT population where feedforward sensory and feedback decision related choice-correlations give rise to a mixed population code. The non-corrupting feedback of choice formation to MT can be useful for tuning of receptive fields and learning of optimal readouts in relation to the task context.

Fig 7.
  • Download figure
  • Open in new tab
Fig 7.

Four possible sources of neural-choice correlation. 1-dimensional stimulus drive to MT is picked up as population variability along with other noise correlations denoted x1(t), x2 (t), x3(t). To optimally perform the task, the choice should rely on only the stimulus dimension, and hence noise in x1 shows up as CP in relevant units reflecting their ‘readout’ strategy (case 1). Non-optimal readout can provide CP through stimulus-irrelevant variability (case 3). Alternatively, feedback from the decision-making process to MT can provide choice-correlation in the stimulus-irrelevant subspace (case 4) without corrupting the optimal representation or the stimulus driven shared dimension (case 2) causing non-optimal behavior.

Acknowledgments

Yuan Zhao was supported by IIS-1734910 from National Science Foundation. Jacob Yates was supported by National Institutes of Health under Ruth L. Kirschstein National Research Service Awards T32DA018926 from the National Institute on Drug Abuse, T3EY021462 and T32EY007125 from National Eye Institute, and is an Open Philanthropy fellow of the Life Sciences Research Foundation. Aaron Levi was supported by National Institutes of Health under Ruth L. Kirschstein National Research Service Awards T32DA018926 from the National Institute on Drug Abuse and T3EY021462 from National Eye Institute. Alexander Huk was supported by IIS-1734910 from National Science Foundation and EY017366 from National Eye Institute. IL Memming Park was supported by IIS-1734910 from National Science Foundation.

References

  1. 1.↵
    Tolhurst DJ, Movshon JA, Dean AF. The statistical reliability of signals in single neurons in cat and monkey visual cortex. Vision Research. 1983;23(8):775–785. doi:10.1016/0042-6989(83)90200-6.
    OpenUrlCrossRefPubMedWeb of Science
  2. 2.↵
    Goris RLT, Ziemba CM, Stine GM, Simoncelli EP, Movshon JA. Dissociation of Choice Formation and Choice-Correlated Activity in Macaque Visual Cortex. The Journal of Neuroscience. 2017;37(20):5195–5203. doi:10.1523/jneurosci.3331-16.2017.
    OpenUrlAbstract/FREE Full Text
  3. 3.↵
    Smith FV, Bird MW. The relative attraction for the domestic chick of combinations of stimuli in different sensory modalities. Animal Behaviour. 1963;11(2-3):300–305. doi:10.1016/s0003-3472(63)80115-3.
    OpenUrlCrossRef
  4. 4.
    Zylberberg A. Neurophysiological bases of exponential sensory decay and top-down memory retrieval: a model. Frontiers in Computational Neuroscience. 2009;3. doi:10.3389/neuro.10.004.2009.
    OpenUrlCrossRef
  5. 5.
    Ponce-Alvarez A, Albright TD, Stoner GR, Deco G. Stimulus-dependent variability and noise correlations in cortical MT neurons. Proceedings of the National Academy of Sciences. 2013;110(32):13162–13167. doi:10.1073/pnas.1300098110.
    OpenUrlAbstract/FREE Full Text
  6. 6.↵
    Cohen MR, Newsome WT. Context-Dependent Changes in Functional Circuitry in Visual Area MT. Neuron. 2008;60(1):162–173. doi:10.1016/j.neuron.2008.08.007.
    OpenUrlCrossRefPubMedWeb of Science
  7. 7.↵
    Bondy AG, Haefner RM, Cumming BG. Feedback determines the structure of correlated variability in primary visual cortex. Nature Neuroscience. 2018;21(4):598–606. doi:10.1038/s41593-018-0089-1.
    OpenUrlCrossRefPubMed
  8. 8.↵
    Britten KH, Newsome WT, Shadlen MN, Celebrini S, Movshon JA. A relationship between behavioral choice and the visual responses of neurons in macaque MT. Visual Neuroscience. 1996;13(01):87–100. doi:10.1017/s095252380000715x.
    OpenUrlCrossRefPubMedWeb of Science
  9. 9.↵
    Nienborg H, Cumming BG. Decision-related activity in sensory neurons reflects more than a neuron’s causal effect. Nature. 2009;459(7243):89–92. doi:10.1038/nature07821.
    OpenUrlCrossRefPubMedWeb of Science
  10. 10.
    Uka T, DeAngelis GC. Linking Neural Representation to Function in Stereoscopic Depth Perception: Roles of the Middle Temporal Area in Coarse versus Fine Disparity Discrimination. Journal of Neuroscience. 2006;26(25):6791–6802. doi:10.1523/jneurosci.5435-05.2006.
    OpenUrlAbstract/FREE Full Text
  11. 11.↵
    Gu Y, Liu S, Fetsch CR, Yang Y, Fok S, Sunkara A, et al. Perceptual Learning Reduces Interneuronal Correlations in Macaque Visual Cortex. Neuron. 2011;71(4):750–761. doi:10.1016/j.neuron.2011.06.015.
    OpenUrlCrossRefPubMedWeb of Science
  12. 12.↵
    Shadlen MN, Newsome WT. Neural Basis of a Perceptual Decision in the Parietal Cortex (Area LIP) of the Rhesus Monkey. Journal of Neurophysiology. 2001;86(4):1916–1936. doi:10.1152/jn.2001.86.4.1916.
    OpenUrlCrossRefPubMedWeb of Science
  13. 13.
    Gold JI, Shadlen MN. Representation of a perceptual decision in developing oculomotor commands. Nature. 2000;404(6776):390–394. doi:10.1038/35006062.
    OpenUrlCrossRefPubMedWeb of Science
  14. 14.↵
    Pitkow X, Liu S, Angelaki DE, DeAngelis GC, Pouget A. How Can Single Sensory Neurons Predict Behavior? Neuron. 2015;87(2):411–423. doi:10.1016/j.neuron.2015.06.033.
    OpenUrlCrossRefPubMed
  15. 15.↵
    Haefner RM, Gerwinn S, Macke JH, Bethge M. Inferring decoding strategies from choice probabilities in the presence of correlated variability. Nature Neuroscience. 2013;16(2):235–242. doi:10.1038/nn.3309.
    OpenUrlCrossRefPubMed
  16. 16.↵
    Haefner RM, Berkes P, Fiser J. Perceptual Decision-Making as Probabilistic Inference by Neural Sampling. Neuron. 2016;90(3):649–660. doi:10.1016/j.neuron.2016.03.020.
    OpenUrlCrossRefPubMed
  17. 17.↵
    Panzeri S, Harvey CD, Piasini E, Latham PE, Fellin T. Cracking the Neural Code for Sensory Perception by Combining Statistics, Intervention, and Behavior. Neuron. 2017;93(3):491–507. doi:10.1016/j.neuron.2016.12.036.
    OpenUrlCrossRefPubMed
  18. 18.↵
    Zohary E, Shadlen MN, Newsome WT. Correlated neuronal discharge rate and its implications for psychophysical performance. Nature. 1994;370(6485):140–143. doi:10.1038/370140a0.
    OpenUrlCrossRefPubMedWeb of Science
  19. 19.↵
    Moreno-Bote R, Beck J, Kanitscheider I, Pitkow X, Latham P, Pouget A. Information-limiting correlations. Nature Neuroscience. 2014;17(10):1410–1417. doi:10.1038/nn.3807.
    OpenUrlCrossRefPubMed
  20. 20.↵
    Wimmer K, Compte A, Roxin A, Peixoto D, Renart A, Rocha Jdl. Sensory integration dynamics in a hierarchical network explains choice probabilities in cortical area MT. Nature Communications. 2015;6:6177. doi:10.1038/ncomms7177.
    OpenUrlCrossRefPubMed
  21. 21.↵
    Lange RD, Haefner RM. Characterizing and interpreting the influence of internal variables on sensory activity. Current opinion in neurobiology. 2017;46:84–89.
    OpenUrl
  22. 22.↵
    Kaufman MT, Churchland MM, Ryu SI, Shenoy KV. Cortical activity in the null space: permitting preparation without movement. Nat Neurosci. 2014;17(3):440–448.
    OpenUrlCrossRefPubMed
  23. 23.↵
    Zhao Y, Park IM. Variational Latent Gaussian Process for Recovering Single-Trial Dynamics from Population Spike Trains. Neural Computation. 2017;29(5):1293–1316. doi:10.1162/neco_a_00953.
    OpenUrlCrossRefPubMed
  24. 24.↵
    Yates JL, Park IM, Katz LN, Pillow JW, Huk AC. Functional dissection of signal and noise in MT and LIP during decision-making. Nature Neuroscience. 2017;20(9):1285–1292. doi:10.1038/nn.4611.
    OpenUrlCrossRefPubMed
  25. 25.↵
    Yu BM, Cunningham JP, Santhanam G, Ryu SI, Shenoy KV, Sahani M. Gaussian-Process Factor Analysis for Low-Dimensional Single-Trial Analysis of Neural Population Activity. Journal of Neurophysiology. 2009;102(1):614–635. doi:10.1152/jn.90941.2008.
    OpenUrlCrossRefPubMedWeb of Science
  26. 26.↵
    Graf ABA, Kohn A, Jazayeri M, Movshon JA. Decoding the activity of neuronal populations in macaque primary visual cortex. Nature Neuroscience. 2011;14(2):239–245. doi:10.1038/nn.2733.
    OpenUrlCrossRefPubMedWeb of Science
  27. 27.↵
    Engel TA, Chaisangmongkon W, Freedman DJ, Wang XJ. Choice-correlated activity fluctuations underlie learning of neuronal category representation. Nat Commun. 2015;6:6454.
    OpenUrlCrossRefPubMed
  28. 28.↵
    Levi AJ, Yates JL, Huk AC, Katz LN. Strategic and Dynamic Temporal Weighting for Perceptual Decisions in Humans and Macaques. eNeuro. 2018;5(5):ENEURO.0169–18.2018.
    OpenUrl
  29. 29.↵
    Katz LN, Yates JL, Pillow JW, Huk AC. Dissociated functional significance of decision-related activity in the primate dorsal stream. Nature. 2016;535(7611):285–288. doi:10.1038/nature18617.
    OpenUrlCrossRefPubMed
  30. 30.↵
    Golub GH, Heath M, Wahba G. Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter. Technometrics. 1979;21(2):215–223. doi:10.1080/00401706.1979.10489751.
    OpenUrlCrossRefWeb of Science
  31. 31.↵
    Lueckmann JM, Macke JH, Nienborg H. Can Serial Dependencies in Choices and Neural Activity Explain Choice Probabilities? The Journal of Neuroscience. 2018;38(14):3495–3506. doi:10.1523/jneurosci.2225-17.2018.
    OpenUrlAbstract/FREE Full Text
  32. 32.↵
    Gao P, Trautmann E, Yu BM, Santhanam G, Ryu S, Shenoy K, et al. A theory of multineuronal dimensionality, dynamics and measurement. bioArxiv. 2017;doi:10.1101/214262.
    OpenUrlAbstract/FREE Full Text
  33. 33.↵
    Shadlen M, Britten K, Newsome W, Movshon J. A computational analysis of the relationship between neuronal and behavioral responses to visual motion. The Journal of Neuroscience. 1996;16(4):1486–1510. doi:10.1523/jneurosci.16-04-01486.1996.
    OpenUrlAbstract/FREE Full Text
  34. 34.
    Cumming BG, Nienborg H. Feedforward and feedback sources of choice probability in neural population responses. Curr Opin Neurobiol. 2016;37:126–132.
    OpenUrlCrossRefPubMed
  35. 35.↵
    Crapse TB, Basso MA. Insights into decision making using choice probability. Journal of Neurophysiology. 2015;114(6):3039–3049. doi:10.1152/jn.00335.2015.
    OpenUrlCrossRefPubMed
View Abstract
Back to top
PreviousNext
Posted December 20, 2019.
Download PDF
Email

Thank you for your interest in spreading the word about bioRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Stimulus-choice (mis)alignment in primate MT cortex
(Your Name) has forwarded a page to you from bioRxiv
(Your Name) thought you would like to see this page from the bioRxiv website.
Share
Stimulus-choice (mis)alignment in primate MT cortex
Yuan Zhao, Jacob L. Yates, Aaron J. Levi, Alexander C. Huk, Il Memming Park
bioRxiv 2019.12.20.884189; doi: https://doi.org/10.1101/2019.12.20.884189
Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
Citation Tools
Stimulus-choice (mis)alignment in primate MT cortex
Yuan Zhao, Jacob L. Yates, Aaron J. Levi, Alexander C. Huk, Il Memming Park
bioRxiv 2019.12.20.884189; doi: https://doi.org/10.1101/2019.12.20.884189

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Neuroscience
Subject Areas
All Articles
  • Animal Behavior and Cognition (1641)
  • Biochemistry (2721)
  • Bioengineering (1902)
  • Bioinformatics (10201)
  • Biophysics (4174)
  • Cancer Biology (3202)
  • Cell Biology (4522)
  • Clinical Trials (135)
  • Developmental Biology (2831)
  • Ecology (4447)
  • Epidemiology (2041)
  • Evolutionary Biology (7213)
  • Genetics (5463)
  • Genomics (6793)
  • Immunology (2379)
  • Microbiology (7462)
  • Molecular Biology (2978)
  • Neuroscience (18529)
  • Paleontology (135)
  • Pathology (472)
  • Pharmacology and Toxicology (776)
  • Physiology (1147)
  • Plant Biology (2692)
  • Scientific Communication and Education (679)
  • Synthetic Biology (885)
  • Systems Biology (2840)
  • Zoology (465)