Abstract
The activity of a neural population encodes information about the stimulus that caused it, and decoding population activity reveals how neural circuits process that information. Correlations between neurons strongly impact both encoding and decoding, yet we still lack models that simultaneously capture stimulus encoding by large populations of correlated neurons and allow for accurate decoding of stimulus information, thus limiting our quantitative understanding of the neural code. To address this, we propose a class of models of large-scale population activity based on the theory of exponential family distributions. We apply our models to macaque primary visual cortex (V1) recordings, and show they capture a wide range of response statistics, facilitate accurate Bayesian decoding, and provide interpretable representations of fundamental properties of the neural code. Ultimately, our framework could allow researchers to quantitatively validate predictions of theories of neural coding against both large-scale response recordings and cognitive performance.
Introduction
A foundational idea in sensory neuroscience is that the activity of neural populations constitutes a “neural code” for representing stimuli (Dayan and Abbott, 2005; Doya, 2007): the activity pattern of a population in response to a sensory stimulus encodes information about that stimulus, and downstream neurons decode, process, and re-encode this information in their own responses. Sequences of such neural populations implement the elementary functions that drive perception, cognition, and behaviour (Pitkow and Angelaki, 2017). Therefore, by studying the encoding and decoding of population responses, researchers may investigate how information is processed along neural circuits, and how this processing influences perception and behaviour (Wei and Stocker, 2015; Panzeri et al., 2017; Kriegeskorte and Douglas, 2018).
Given a true statistical model of how a neural population responds to (encodes information about) stimuli, Bayes’ rule can transform the encoding model into an optimal decoder of stimulus information (Zemel et al., 1998; Pillow et al., 2010). However, when validated as Bayesian decoders, existing statistical models of neural encoding are often outperformed by models trained to decode stimulus-information directly, indicating that the encoding models miss key statistics of the neural code (Graf et al., 2011; Walker et al., 2020). In particular, the correlations between neurons’ responses to repeated presentations of a given stimulus (noise correlations), and how these noise correlations are modulated by stimuli, can strongly impact coding in neural circuits (Zohary et al., 1994; Abbott and Dayan, 1999; Sompolinsky et al., 2001; Ecker et al., 2016; Kohn et al., 2016; Schneidman, 2016), especially in large populations of neurons (Moreno-Bote et al., 2014; Montijn et al., 2019; Bartolo et al., 2020; Kafashan et al., 2020; Rumyantsev et al., 2020). Yet effectively modelling noise correlations has proven challenging.
Validating theories of population coding (Ma et al., 2006; Beck et al., 2011a; Ganguli and Simoncelli, 2014; Makin et al., 2015; Yerxa et al., 2020) in large neural circuits thus depends on encoding models that support accurate Bayesian decoding, effectively capture noise-correlations, and eiciently fit large-scale neural recordings. Generalized linear models (GLMs) are one class of model that yield effective Bayesian decoders, and GLMs have been applied to analyzing spatio-temporal features of information processing in the retina and cortex (Pillow et al., 2008; Park et al., 2014; Runyan et al., 2017). Nevertheless, neural correlations are often the result of low-dimensional, shared variability (Arieli et al., 1996; Ecker et al., 2014; Goris et al., 2014; Rabinowitz et al., 2015; Okun et al., 2015; Semedo et al., 2019), and it is unknown whether extensions of the GLM approach to capture shared-variability (Archer et al., 2014; Zhao and Park, 2017) can support accurate Bayesian decoding. Similarly, methods based on factor analysis (Yu et al., 2009; Ecker et al., 2014; Semedo et al., 2019) have proven highly effective at modelling neural correlations in largescale recordings, but it also unknown if they can support Bayesian decoding. Finally, a model class related to GLMs is pairwise-maximum entropy models (Schneidman et al., 2006; Lyamzin et al., 2010; Granot-Atedgi et al., 2013; Meshulam et al., 2017), which have been used to investigate semantic clustering of responses in the retinal code (Ganmor et al., 2015); yet these models have so-far been limited to population sizes of tens of neurons.
Towards modelling responses and accurate Bayesian decoding in large populations of correlated neurons, we have developed a class of spike-count encoding model based on conditional finite mixtures of multivariate Poisson distributions, which we refer to as CPMs (Conditional Poisson Mixtures). Within neuroscience, Poisson mixtures are widely applied to modelling the spike-count distributions of individual neurons (Wiener and Richmond, 2003; Shidara et al., 2005; Goris et al., 2014; Taouali et al., 2015). Outside of neuroscience, mixtures of multivariate Poisson distributions are an established model of multivariate count distributions that effectively capture correlations in count data (Karlis and Meligkotsidou, 2007; Inouye et al., 2017).
Building on the theory of exponential family distributions (Wainwright and Jordan, 2008; Macke et al., 2011b), our model extends previous mixture models of multivariate count data in two ways. Firstly, we develop a tractable extension of Poisson mixtures that captures both over-and underdispersed response variability (i.e. where the response variance is larger or smaller than the mean, respectively) based on Conway-Maxwell Poisson distributions (Shmueli et al., 2005; Stevenson, 2016). Secondly, we introduce an explicit dependence of the model on a stimulus variable, which allows the model to accurately capture changes in response statistics (including noise correlations) across stimuli. Importantly, the resulting encoding model affords closed-form expressions for both its Fisher information and probability density function and thereby a rigorous quantification of the coding properties of a modelled neural population (Dayan and Abbott, 2005). Moreover, the model learns low-dimensional representations of stimulus-driven neural activity, and we show how it provides a parsimonious description of a fundamental property of population codes known as information-limiting correlations (Moreno-Bote et al., 2014; Montijn et al., 2019; Bartolo et al., 2020; Kafashan et al., 2020; Rumyantsev et al., 2020).
We apply the CPM framework to both synthetic data and recordings from macaque primary visual cortex (V1), and demonstrate that it effectively models responses of populations of hundreds of neurons, captures noise correlations, and supports accurate Bayesian decoding. Ultimately, our model of neural encoding and decoding can be used to quantify coding properties of a neural circuit, such as their efficiency, linearity, or information capacity.
Results
A critical part of our theoretical approach is based on expressing models of interest in exponential family form. An exponential family distribution p(n) over some data n (in our case, neural responses) is defined by the proportionality relation p(n) ∝ eθ·S(n)b(n), where θ are the so-called natural parameters, S(n) is a vector-valued function of the data called the sufficient statistic, and b(n) is a scalar-valued function called the base measure (Wainwright and Jordan, 2008). The exponential family form allows us to modify and extend existing models in a simple and flexible manner, and to gain analytical insight into the coding properties of our models. We demonstrate our approach with applications to both synthetic data generated by example CPMs, and data recorded in V1 of anaesthetized and awake macaques viewing drifting grating stimuli at different orientations (for details see Materials and methods).
Extended Poisson mixture models capture spike-count variability and covariability
Our first goal is to define a class of models of neural population activity, that model neural activity directly as spike-counts, and that accurately capture single-neuron variability and pairwise covariability. We base our models on Poisson distributions, as they are widely-applied to modelling the trial-to-trial distribution of the number of spikes generated by a neuron (Dayan and Abbott, 2005; Macke et al., 2011a). We will also generalize our Poisson models with Conway-Maxwell (CoM) Poisson distributions, because they can capture the broad range of Fano factors (FF; the variance divided by the mean) observed in cortex, in contrast with Poisson distributions for which the FF is always 1 (Sur et al., 2015; Stevenson, 2016; Chanialidis et al., 2018).
Mixtures of Poisson distributions are also used to capture complex spike-count distributions in cortex, and allow for over-dispersion (FF>1) (Shidara et al., 2005; Goris et al., 2014; Taouali et al., 2015) (Figure 1A). In our case we consider multivariate Poisson mixtures, as they capture covariability in count data as well (see Karlis and Meligkotsidou (2007) for the general definition). To construct a multivariate Poisson mixture we begin with a product of independent Poisson distributions, one per neuron. We then mix a finite number of such independent Poisson models, to arrive at a multivariate spike-count, finite mixture model (see Materials and methods). Importantly, although each mixture component is a product of independent Poisson distributions, randomly switching between components induces correlations between the neurons (Figure 1B,C). In fact, multivariate Poisson mixtures may model arbitrary pairwise covariability (see Materials and methods, Equation 6). Nevertheless, they are limited because the variance of individual neurons cannot be smaller than the mean, and are thus always over-dispersed (Equation 5, Materials and methods).
To address this limitation, we show how to express multivariate Poisson mixtures in an exponential family form, and then generalize the model with CoM-Poisson distributions. We first note that a multivariate Poisson mixture with dK components may be expressed as a latent variable model over spike-count vectors n and latent component-indices k, where 1 ≤ k ≤ dk. In this formulation we denote the kth component distribution by p(n|k), and the probability of realizing (switching to) the kth component by p(k). The mixture model over spike-counts n is then expressed as the marginal distribution , of the joint distribution p(n,k). Under mild regularity assumptions (see Materials and methods), we may reparameterize this joint distribution in exponential family form as where θN, θK, and ΘNK are the natural parameters of p(n, k), and is the Kronecker delta vector defined by δJ (k) = 1 if J = k, and 0 otherwise.
The exponential family form of a multivariate Poisson mixture represents the first component distribution (i.e. p(n | k) with index k = 1) as a baseline distribution, and the other components (where k > 1) as modulations of the baseline distribution, and this representation helps us extend multivariate Poisson mixtures. In particular, the first component distribution has natural (baseline) parameters θN, and for k > 1, the natural parameters of p(n | k) are the sum of the baseline parameters θN and one row from the matrix of parameters ΘNK (Equation 12, Materials and methods). Because the dimension of θN is much smaller than the total number of parameters in a given mixture, the baseline parameters provide a relatively low-dimensional means of affecting all the component distributions of the given mixture, as well as the index probabilities (Figure 1D; see Materials and methods, Equation 11 for how p(k) depends on θN).
We now extend Relation 1 with CoM-Poisson theory, and propose the latent variable exponential family Where is the vector of log-factorials of the individual spike-counts, and are a set of natural parameters based on CoM-Poisson distributions (see Materials and methods). The exponential family form continues to represent the mixture in terms of a baseline distribution, in this case p(n | k) is a product of independent CoM-Poisson distributions, with baseline parameters θN and CoM-based parameters . However, whereas the rows of ΘNK modulate θN depending on the component index k, the parameters are not modulated, and remain the same for each component distribution (Equation 15, Materials and methods, and see Equation 14 for index-probability formula). For the rest of this paper we refer to models described by Relation 1 as vanilla mixtures, and models described by Relation 2 as CoM-based mixtures.
Due to the addition of the CoM-based parameters, a CoM-based mixture can model underdispersed (FF < 1) neural activity (Equation 16, Materials and methods). In Figures 1D-F we demonstrate how changing the parameters of the CoM-based mixture can concentrate or disperse both the mixture distribution and its components.
To validate our mixture models, we tested if they capture variability and covariability of V1 population responses to repeated presentations of a grating stimulus with fixed orientation (dN = 43 neurons and dT = 355 repetitions in one awake macaque; dN = 70 and dT = 1, 200 in one anaesthetized macaque). We optimized model parameters as described in Materials and methods. The CoM-Poisson mixture accurately captured single-neuron variability (Figure 2A-B, red symbols), including both cases of over-dispersion and under-dispersion. In contrast, the simpler multivariate Poisson mixture (Figure 2A-B, blue symbols) could not accommodate under-dispersion, and also had a limited ability to model over-dispersion due to the coupling between the mean and variance (Equation 5). On the other hand, we found that both mixture models were flexible enough to qualitatively capture pairwise noise correlations, both in awake and anaesthetized animals (Fig. 2C-H) (in later sections we quantitatively compare the model performance).
Extended Poisson mixture models capture stimulus-dependent response statistics
So far we have introduced the exponential family theory of vanilla and CoM-based Poisson mixtures, and shown how they capture response variability and covariability for a fixed stimulus. To allow us to study stimulus encoding and decoding, we further extend our mixtures by inducing a dependency of the model parameters on a stimulus. When there are a finite number of stimulus conditions and sufficient data, we may define a stimulus-dependent model with a lookup table, and fit it by fitting a distinct model at each stimulus condition. However, this is inefficient when the amount of data at each stimulus-condition is limited and the stimulus-dependent statistics have structure that is shared across conditions. A notable feature of the exponential family parameterizations in Relations 1 and 2 is that the baseline parameters influence both the index probabilities and all the component distributions of the model. This suggests that by restricting stimulus-dependence to the baseline parameters, we might model rich stimulus-dependent response structure, while bounding the complexity of the model.
In general we refer to any finite mixture of independent Poisson distributions with stimulus-dependent parameters as a conditional Poisson mixture (CPM), and depending on whether the CPM is based on Relations 1 or 2, we refer to it as a vanilla or CoM-based CPM, respectively. Although there are many ways we might induce stimulus-dependence, in this paper we consider two forms of CPM: (i) a maximal CPM, which we implement as a lookup table, such that all the parameters in Relation 1 or 2 depend on the stimulus, and (ii) a minimal CPM, for which we restrict stimulus-dependence to the baseline parameters θN, resulting in the CoM-based CPM where x is the stimulus, and θN (x) are the stimulus-dependent baseline parameters (we may recover a minimal, vanilla CPM by setting ). The tuning curves of the CPM neurons are the average spike-counts (firing rates) of each ni as a function of the stimulus x, and we refer to θN (x) as the baseline tuning curve parameters, as they define how the firing rates of the baseline CPM distribution (i.e. p(n | x, k) when k = 1) depend on x. For k > 1, the modulated CPM p(n | x, k) is then a scaled, or “gain-modulated” version of the baseline CPM (see Equations 12 and 15 and the accompanying discussions).
Towards understanding the expressive power of CPMs, we study a minimal, CoM-based CPM with dN = 20 neurons, dK = 5 mixture components, and randomly chosen parameters (see Materials and methods). Moreover, we assume that the stimulus is periodic (e.g. the orientation of a grating), and that the baseline tuning curves have a von Mises shape which is a widely applied model of neural tuning to periodic stimuli (Herz et al., 2017). We may achieve such a baseline shape by defining the baseline tuning curve parameters as , where and ΘNX are the tuning curve parameters, and vm(x) = (cos 2x, sin 2x). Figure 3A shows that the tuning curves of the CPM neurons are approximately bell-shaped, yet many also exhibit significant deviations.
We also study if CPMs can be effectively fit to datasets comparable to those obtained in typical neurophysiology experiments. We generated 200 responses from the CoM-based CPM described above — the ground truth CPM — to each of 10 orientations spread evenly over the half-circle, for a total of 2,000 stimulus-response sample points. We then used this data to fit a CPM with the same number of components. Towards this aim, we derived an approximate expectation-maximization algorithm (EM, a standard choice for training finite mixture models (McLachlan et al., 2019)) to optimize model parameters, that also accounts for the stimulus-dependence (see Materials and methods). Figure 3B shows that the tuning curves of the learned CPM are nearly indistinguishable from those of the ground truth CPM (Figure 3B).
To reveal the orientation-dependent latent structure of the model, in Figure 3C we plot the index probability p(k | x) for every k as a function of the orientation x. In Figure 3D we show that the orientation-dependent index probabilities of the learned CPM qualitatively match the true index probabilities in Figure 3C. We also note that although the learned CPM does not correctly identify the indices themselves, this has no effect on the performance of the CPM.
The orientation-dependent index-probabilities provide a high-level picture of how the complexity and structure of model correlations varies with the orientation. The vertical dashed lines in Figures 3C-D denote two orientations that yield substantially different index probabilities p(k | x). When a large number of index-probabilities are non-zero, the correlation-matrices of the CoM-based CPM can exhibit complex correlations with both negative and positive values (Figure 3E). However, when one index dominates, the correlation structure largely disappears (Figure 3F). In Figure 3G we show that the FFs also depend on stimulus orientation. Lastly, we find that both the FF and the correlation-matrices of the learned CPM are nearly indistinguishable from the groundtruth CPM (Figure 3E-G).
In summary, our analyses show that CPMs can generate complex, stimulus-dependent response statistics, and that the learned CPM accurately recovers both the statistics and the latent structure of the neural responses from realistic amounts of data.
CPMs effectively model neural responses in macaque V1
A variety of models may be defined within the CPM framework illustrated by Relations 1, 2, and 3. Towards understanding how effectively CPMs can model real data, we compare different variants by their cross-validated log-likelihood. We consider both vanilla and CoM-based variants of each of the following conditional mixtures: (i) maximal CPMs where we learn a distinct mixture for each of dX stimulus conditions, (ii) minimal CPMs with von Mises baseline tuning curves, and (iii) minimal CPMs with discrete baseline tuning curves given by , where δ is the Kronecker delta vector with dX − 1 elements, and x is the index of the stimulus. In contrast with the von Mises CPM, the discrete CPM makes no assumptions about the form of baseline tuning.
To provide an interpretable measure of the relative performance of each CPM variant, we measured the difference between the estimated log-likelihood of the given CPM and the log-likelihood of a von Mises-tuned, independent Poisson model, which is a standard model of uncorrelated neural responses to oriented stimuli (Herz et al., 2017). We refer to this quantity as the information gain.
Table 1 shows that the CPM variants considered achieve comparable performance, and perform substantially better than the independent Poisson lower bound on both the awake and anaesthetized data. Figure 4 shows that a performance peak emerges smoothly as the model complexity (number of parameters) is increased. In all cases, the CoM-based models outperform their vanilla counterparts, and typically with fewer parameters. The CoM-based discrete CPMs achieve high performance on both datasets. In contrast, von Mises CPMs perform well on the anaesthetized data but more poorly on the awake data, and maximal CPMs exhibit the opposite trend. Nevertheless, von Mises CPMs solve a more difficult statistical problem as they also interpolate between stimulus conditions, and so may still prove relevant even where performance is limited. On the other hand, even though maximal CPMs achieve high performance, they simply do so by replicating the high performance of stimulus-independent mixtures (Figure 2) at each stimulus condition, requiring significantly more parameters than minimal CPMs.
CPMs facilitate accurate and efficient Bayesian decoding of neural responses
To demonstrate that CPMs model the neural code, we must show that CPMs not only capture the features of neural responses, but that these features also encode stimulus-information. Given an encoding model p(n | x) and a response from the model n, we may optimally decode the information in the response about the stimulus x by applying Bayes’ rule p(x | n) ∝ p(n | x)p(x), where p(x | n) is the posterior distribution (the decoded information), and p(x) represents our prior assumptions about the stimulus (Zemel et al., 1998). When we do not know the true encoding model, and rather fit a statistical model to stimulus-response data, using the statistical model for Bayesian decoding and analyzing its performance can tell us how well it captures the features of the neural code.
We analyze the performance of Bayesian decoders based on CPMs by quantifying their decoding performance, and comparing the results to other common approaches to decoding. We quantify decoding performance by evaluating the average of the cross-validated log-posterior probability log p(x | n) of the true stimulus value x, for both our awake and anaesthetized V1 datasets. With regards to training the CPMs, we analyze the decoding performance of CPMs that achieved the best encoding performance based as indicated in Table 1 and depicted Figure 4, instead of applying distinct procedures for selecting CPMs based on decoding performance. This is because our goal is to understand how well the response features captured by CPMs reflect the neural code, rather than strictly maximizing decoding performance.
In our comparisons we focus on minimal, discrete CPMs as overall they achieved high performance on both datasets (Figure 4). To characterize the importance of neural correlations to Bayesian decoding, we compare our CPMs to the decoding performance of independent Poisson models with discrete tuning (IP). To characterize the optimality of our Bayesian decoders, we also evaluate the performance of linear multiclass decoders (Linear), as well nonlinear multiclass decoders defined as artificial neural networks (ANNs) with two hidden layers and a cross-validated number of hidden units (for details on the training and model selection procedure, see Materials and methods).
Table 2 shows that on the awake data, the performance of the CPMs is statistically indistinguishable from the ANN, and the CPMs and the ANN significantly exceed the performance of both the Linear and IP models. On the anaesthetized data, the minimal CPM approaches the performance of the ANN, and the minimal CPMs and ANN models again exceed the performance of the IP and Linear models. Yet in this case the Linear model is much more competitive, whereas the IP model performs very poorly, possibly because of the larger magnitude of noise correlations in this data. In both cases the ANN requires two orders of magnitude more parameters than the CPMs to achieve its performance gains. In addition, the CoM-based CPM achieves marginally better performance with fewer parameters than the vanilla CPM, indicating that although modelling individual variability is not essential for effective Bayesian decoding, doing so still results in a more parsimonious model of the neural code.
We also consider widely used alternative measures of decoding performance, namely the Fisher information (FI), which is an upper bound on the average precision (inverse variance) of the posterior (Brunel and Nadal, 1998), as well as the linear Fisher information (LFI), which is a linear approximation of the FI (Seriès et al., 2004) corresponding to the accuracy of the optimal, unbiased linear decoder of the stimulus (Kanitscheider et al., 2015a). The FI is especially helpful when the posterior cannot be evaluated directly (such as when it is continuous), and is widely adopted in theoretical (Abbott and Dayan, 1999; Ecker et al., 2014; Moreno-Bote et al., 2014; Kohn et al., 2016) and experimental (Ecker et al., 2011; Rumyantsev et al., 2020) studies of neural coding. As with other models based on exponential family theory (Ma et al., 2006; Beck et al., 2011b; Ecker et al., 2016), the FI of a minimal CPM may be expressed in closed-form, and is equal to its LFI (see Materials and methods), and therefore minimal CPMs can be used to study FI analytically and obtain model-based estimates of FI from data.
We generated 40 populations of dN = 20 model neurons from the vanilla, minimal, von Mises CPM, with parameters corresponding to the best-fit parameters of 40 random subsets of neurons from our V1 datasets. For each population, we generated 50 responses at each of 10 evenly spaced orientations, for a total of dT = 500 responses per population. We then fit a CPM to each set of 500 responses, and compared the FI of the fit CPM to the ground-truth FI at 50 evenly spaced orientations. Pooled over all populations and orientations, the relative error of the estimated FI was −12.8% ± 18.6% on the awake data and −9.1% ± 22.4% on the anaesthetized data.
The aforementioned measures allow us to assess decoding performance when we do not know the full posterior, however the full posterior is an essential part of probabilistic neural codes (Pouget et al., 2016; Drugowitsch et al., 2019). To test whether CPMs can in principle recover full posteriors, we consider a ground truth CPM defined as discrete, CoM-based, minimal CPM with dN = 200 neurons, dS = 20 stimulus-conditions, dK = 30 components, and randomized parameters, and we fit a discrete, CoM-based, minimal CPM with dK = 40 components (chosen with cross-validation) to dT = 10, 000 responses from the ground-truth CPM (see Materials and methods). We then compute the average KL-divergence (a fundamental measure of the similarity of two distributions, see Cover and Thomas (2006); Amari and Nagaoka (2007)) of the learned posteriors from the ground-truth posterior over all the dT = 10, 000 responses, and find that the average posterior divergence is 0.047 ± 0.007 bits, indicating that on average the learned and ground-truth posteriors are extremely close.
To summarize, CPMs support accurate Bayesian decoding in awake and anaesthetized macaque V1 recordings, and are competitive with nonlinear decoders with two orders of magnitude more parameters. Moreover, CPMs afford closed-form expressions of FI and can interpolate good estimates of FI from modest amounts of data, and thereby support analyses of neural data based on this widely applied theoretical tool. Finally, a CPM fit to the responses of a ground-truth CPM can almost perfectly recover the ground-truth posterior distributions.
Minimal CPMs provide an interpretable latent representation of a fundamental feature of the neural code
Having shown that CPMs can be used to accurately decode stimuli, we next aim to demonstrate that the latent structure of CPMs offers an interpretable representation of a central phenomenon in neural coding known as information-limiting correlations, which are neural correlations that fundamentally limit stimulus-information in neural circuits (Moreno-Bote et al., 2014; Montijn et al., 2019; Bartolo et al., 2020; Kafashan et al., 2020; Rumyantsev et al., 2020). To illustrate this, we generate population responses with limited information, and then fit a CPM to these responses and study the learned latent representation. In particular, we consider a source population of 200 independent Poisson neurons p(n | s) with homogeneous, von Mises tuning curves responding to a noisy stimulus-orientation s, where the noise p(s | x) follows a von Mises distribution centred at the true stimulus-orientation x (see Materials and methods). In Figure 5A we show that, as expected, the average FI in the source population about the noisy orientation s grows linearly with the size of randomized subpopulations, whereas the FI about the true orientation x is theoretically bounded by the precision (inverse variance) of the sensory noise.
Even though the neurons in the source model are uncorrelated, sensory noise ensures that the information-limited encoding model contains information-limiting correlations that bound the FI about x (Moreno-Bote et al., 2014; Kanitscheider et al., 2015b). To understand whether and how the latent structure of CPMs captures information-limiting noise correlations, we fit a minimal, von Mises, vanilla CPM with dK = 20 mixture components to dT = 10, 000 responses from p(n | x). Figure 5A (purple) shows that the FI of the learned CPM saturates near the precision of the sensory noise, indicating that the learned CPM accurately captures the information-limiting correlations present in p(n | x).
To understand how the learned CPM represents the correlations in p(n | x) we study the relation between the latent modulations and the population activity. Figure 5B shows the index-probabilities of the learned CPM: given the true orientation x = 90°, there are 3 modulations with probabilities substantially greater than 0. To provide a high-level picture of how these modulations affect population responses, in Figure 5C we plot the sum of the modulated rates of the population as a function of orientation, and see that each modulation concentrates the tuning of the population around a particular orientation, and that two of the modulations in particular shift the tuning away from the true orientation.
Because there are essentially three modulations that are relevant to the responses of the CPM to the true orientation x = 90°, generating a response from the CPM approximately reduces to generating a response from one of the three possible modulated populations. In Figures 5D-F we depict a response to x = 90° from each of the three modulated populations, as well as the optimal posterior based on the learned CPM (purple lines), and a suboptimal posterior based on the source model (i.e. ignoring noise correlations; green lines). We observe that the trial-to-trial variability of the learned CPM results in random shifts of the peak neural activity away from the true orientation, thus fundamentally limiting information. Furthermore, when the response of the population is concentrated at the true orientation (Figure 5E), the suboptimal posterior assigns a high probability to the true orientation, whereas when the responses are biased away from the true orientation (Figures 5D and 5F) the suboptimal posterior assigns nearly 0 probability to the true orientation. This is in contrast to the optimal posterior, which always assigns a significant probability to the true orientation.
In summary, CPMs accurately capture information-limiting correlations, and provide insight into how such correlations can be generated by a simple latent structure.
Discussion
In this paper we introduced a latent variable exponential family formulation of multivariate Poisson mixtures. We showed how this formulation allows us to effectively extend multivariate Poisson mixtures both to capture sub-Poisson variability, and to incorporate stimulus dependence, which we termed Conditional Poisson Mixtures (CPMs). Our analyses and simulations showed that CPMs can be fit efficiently and recover ground truth models in synthetic data, capture a wide range of V1 response statistics in real data, and can be easily inverted to obtain accurate Bayesian decoding that is competitive with nonlinear decoders, while using orders of magnitude less parameters. In addition, we illustrated how the latent structure of CPMs provides an interpretable representation of a fundamental feature of the neural code, e.g. information-limiting correlations.
Our framework is particularly relevant for probabilistic theories of neural coding based on the theory of exponential families (Beck et al., 2007), which include theories that address the linearity of Bayesian inference in neural circuits (Ma et al., 2006), the role of phenomena such as divisive normalization in neural computation (Beck et al., 2011a), Bayesian inference about dynamic stimuli (Makin et al., 2015; Sokoloski, 2017), and the metabolic efficiency of neural coding (Ganguli and Simoncelli, 2014; Yerxa et al., 2020). These theories have proven difficult to validate quantitatively with neural data due to a lack of statistical models which are both compatible with their exponential family formulation, and can model correlated activity in recordings of large neural populations. Our work suggests that CPMs can overcome these difficulties, and help connect the rich mathematical theory of neural coding with the state-of-the-art in parallel recording technologies.
CPMs are not limited to modelling neural responses to stimuli, and can model how arbitrary experimental variables modulate neural variability and covariability. Examples of experimental variables that have measurable effects on neural covariability include the spatial and temporal context around a stimulus (Snyder et al., 2014; Snow et al., 2016, 2017; Festa et al., 2020), as well as task-variables and the attentional state of the animal (Maunsell, 2015; Rabinowitz et al., 2015; Kanashiro et al., 2017; Bondy et al., 2018; Ruff and Cohen, 2019). Each of these variables could be incorporated into a CPM by either replacing the stimulus-variable in our equations, or combining it with the stimulus-variable to construct a CPM with multivariate dependence. This would allow researchers to explore how the stimulus and the experimental variables mutually interact to shape variability and covariability in large populations of neurons.
To understand how this variability and covariability effects neural coding, latent variable models such as CPMs are often applied to extract interpretable features of the neural code from data (Whiteway and Butts, 2019). The latent states of a CPM provide a soft classification of neural activity, and we may apply CPMs to model how an experimental variable modulates the class membership of neurons. In the aforementioned studies, models of neural activity yielded predictions of perceptual and behavioural performance. Because CPMs support Bayesian decoding, an appropriate CPM can also make predictions about how a class of neurons is likely to modulate perception and behaviour, and we may then test these predictions with experimental interventions on the neurons themselves (Panzeri et al., 2017). In this manner, we believe CPMs could form a critical part of a rigorous, Bayesian framework for “cracking the neural code” in large populations neurons.
In our applications we considered low-dimensional variables, and implemented the stimulus-dependence of the CPM parameters with linear functions. Nevertheless, the stimulus-dependence of a CPM can be implemented by arbitrary parametric functions of high-dimensional variables such as deep neural networks, and CPMs can also incorporate history-dependence via recurrent neural networks. As such, CPMs have the potential to integrate encoding models of higher cortical areas (Yamins et al., 2014) with models of the temporal features of the neural code (Pillow et al., 2008; Park et al., 2014; Runyan et al., 2017), towards analyzing the neural code in dynamic, correlated neural populations in higher cortex. Outside of neuroscience, high-dimensional count data exists in many fields such as corpus linguistics and genomics (Inouye et al., 2017), and researchers who aim to understand how this data depends on history or additional variables could benefit from our techniques.
Materials and methods
Notation
We use capital, bold letters (e.g. Θ) to indicate matrices; small, bold letters (e.g. θ) to indicate vectors; and regular letters (e.g. θ) to indicate scalars. We use subscript capital letters to indicate the role of a given variable, so that, in Relation 1 for example, θK are the natural parameters that bias the index-probabilities, θN are the baseline natural parameters of the neural firing rates, and ΘNK is the matrix of parameters through which the indicies and rates interact.
We denote the ith element of a vector θ by θi, or e.g. of the vector θK by θK,i. We denote the ith row or Jth column of Θ by θi or θJ, respectively, and always state whether we are considering a row or column of the given matrix. When referring to the Jth element of a vector θi indexed by i, we write θij. Finally, when indexing data points from a sample, or parameters that are tied to individual data points, we use parenthesized, superscript letters, e.g. x(i), or
Poisson mixtures and their moments
The following derivations were presented in a more general form in Karlis and Meligkotsidou (2007), but we present the simpler case here for completeness. A Poisson distribution has the form, where n is the count and λ is the rate (in our case, spike count and firing rate, respectively). We may use a Poisson model to define a distribution over dN spike counts by supposing that the neurons generate spikes independently of one another, leading to the independent Poisson model with firing rates . Finally, if we consider the dK rate vectors , and dK weights , where 0 ≤ w for all k, and , we then define a mixture of Poisson distributions as a latent variable model p(n) = Σkp(n | k)p(k) = Σkp(n, k), where p(n | k) = p(n; λk), and p(k) = wk.
The mean μi of the ith neuron of a mixture of independent Poisson distributions is
The variance of neuron i is Where is the variance of the ith neuron under the kth component distribution, i.e. the variance of p(ni | k), and where , and both follow from the fact that a distribution’s variance equals the difference between its second moment and squared first moment.
The covariance between spike-counts n and n for i ≠ J is then
Observe that if , then is simply the sample covariance between i and J, where the sample is composed of the rate components of the ith and Jth neurons. Equation 6 thus implies that Poisson mixtures can model arbitrary covariances. Nevertheless, Equation 5 shows that the variance of individual neurons is restricted to being larger than their means.
Exponential family mixture models
In this section we show that the latent variable form for Poisson mixtures we introduced above is a member of the class of models known as exponential families. An exponential family distribution p(x) over some data x has the form p(x) = eθ·S(x)−χ(θ)b(x), where θ are the so-called natural parameters, S(x) is a vector-valued function of the data called the sufficient statistic, b(x) is a scalar-valued function called the base measure, and is the log-partition function (Wainwright and Jordan, 2008). In the context of Poisson mixture models, we note that an independent Poisson model p(n; λ) is an exponential family, with natural parameters θN given by θN,i = log λi, base measure and sufficient statistic SN (n) = n, and log-partition function . Moreover, the distribution of component indices p(k) (also known as a categorical distribution) also has an exponential family form, with natural parameters for 1 ≤ k < d, sufficient statistic , base measure b(k) = 1, and log-partition function . Note that in both cases, the exponential parameters are well-defined only if the rates and weights are strictly greater than 0 — in practice however this is not a significant limitation.
We claim that the joint distribution of a multivariate Poisson mixture model p(n, k) can be reparameterized in the exponential family form Where is the log-partition function of p(n | k). To show this we show how to express the natural parameters θN, θK, and ΘNK as (invertible) functions of the component rate vectors , and the weights . In particular, we set where log is applied element-wise. Then, for 1 ≤ k < dk, we set the kth row θN K, k of ΘN K to and the kth element of θK to
This reparameterization may then be checked by substituting Equations 8, 9, and 10 into Equation 7 to recover the joint distribution of the mixture model p(n, k) = p(n | k)p(k) = wkp(n; λK); for a more explicit derivation see Sokoloski (2019).
The equation for p(n, k) ensures that the index-probabilities are given by
Consequently, the component distributions in exponential family form are given by
Observe that p(n | k) is a multivariate Poisson distribution with parameters θN + ΘNK · δ(k), so that for k > 1, the parameters are the sum of θN and row k − 1 of ΘNK. Because the exponential family parameters are the logarithms of the firing rates of n, each row of ΘNK modulates the firing rates of n multiplicatively. When θN (x) depends on a stimulus and we consider the component distributions p(n | x, k), each row of ΘNK then scales the tuning curves of the baseline population (i.e. (p(n | x, k) for k = 1); in the neuroscience literature, such scaling factors are typically referred to as gain modulations.
The exponential family form has many advantages. However, it has a less intuitive relationship with the statistics of the model such as the mean and covariance. The most straightforward method to compute these statistics given a model in exponential family form is to first reparameterize it in terms of the weights and component rates, and then evaluate Equations 4, 5, and 6.
CoM-Poisson distributions and their mixtures
Conway-Maxwell (CoM) Poisson distributions decouple the location and shape of count distributions (Shmueli et al., 2005; Stevenson, 2016; Chanialidis et al., 2018). A CoM Poisson model has the form. The floor function λ of the location parameter λ is the mode of the given distribution. With regards to the shape parameter v, p(n; λ, v) is a Poisson distribution with rate λ when v = 1, and is under- or over-dispersed when v > 1 or v < 1, respectively. A CoM-Poisson model p(n; λ, v) is also an exponential family, with natural parameters θC = (v log λ, −v), sufficient statistic SC (n) = (n, log n!), and base measure b(n) = 1. The log-partition function does not have a closed-form expression, but it can be effectively approximated by truncating the series (Shmueli et al., 2005). More generally, when we consider a product of independent CoM-Poisson distributions, we denote its log-partition function by , where are the parameters of the ith CoM-Poisson distribution. In this case we can also approximate the log-partition function ψC by truncating the dN constituent series in parallel.
We define a multivariate CoM-based mixture as Where is the vector of log-factorials of the individual spike-counts, and is the log-partition function. This form ensures that the index-probabilities satisfy and consequently that each component distribution p(n | k) is a product of independent CoM Poisson distributions given by
Observe that, whereas the parameters θN + ΘNK · δ(k) of p(n | k) depend on the index k, the parameters are independent of the index and act exclusively as biases. Note as well that when considering a CoM-based, minimal CPM, the modulated populations (p(n | k, x) for k > 1) continue to scale the firing rates of the baseline population (p(n | k, x)) monotonically, but not in a linear, multiplicative manner.
The moments of a CoM-Poisson distribution are not available in closed-form, yet they can also be effectively approximated through truncation. Given approximate means μik and variances of p(ni | k), we may easily evaluate the means, variances, and covariances of p(ni). In particular, the mean of ni is , and its variance is Where . Finally, similarly to Equation 6, the covariance σij between ni and nJ is .
By comparing Equations 5 and 16, we see that the CoM-based mixture may address the limitations on the variances of the vanilla mixture by setting the average variance of the components in Equation 16 to be small, while holding the value of the means μi fixed, and ensuring that the means of the components μik cover a wide range of values to achieve the desired values of and σij. Solving the parameters of a CoM-based mixture for a desired covariance matrix is unfortunately not possible since we lack closed-form expressions for the means and variances. Nevertheless, we may justify the effectiveness of the CoM-based strategy by considering the approximations of the components means and variances and , which hold when neither λik or vik are too small (Chanialidis et al., 2018). Based on these approximations, observe that when vik is large, is small, whereas μik is more or less unaffected. Therefore, in the regime where these approximations hold, a small value for can be achieved by reducing the parameters vik, without significantly restricting the values of μik or μi.
Fisher information of a CPM
The Fisher information (FI) of an encoding model p(n | x) with respect to x is I(x) = Σnp(n|x)(∂x log p(n | x))2 (Cover and Thomas, 2006). With regards to the FI of a CPM, where follows from the chain rule and properties of the log-partition function (Wainwright and Jordan, 2008). Therefore where ΣN (x) is the covariance matrix of p(n | x). Moreover, because (Wain-wright and Jordan, 2008), the FI of a CPM may also be expressed as , which is the linear Fisher information (Beck et al., 2011b).
Note that when calculating the FI or other quantities based on the covariance matrix, vanilla CPMs have the advantage that their covariance matrices tend to have large diagonal elements and are thus inherently well-conditioned. Because decoding performance is not significantly different between vanilla and CoM-based CPMs (see Table 2), vanilla CPMs may be preferable when well-conditioned covariance matrices are critical. Nevertheless, the covariance matrices of CoM-based mixtures can be made well-conditioned by applying standard techniques.
Expectation-Maximization for CPMs
Expectation-maximization (EM) is an algorithm that maximizes the likelihood of a latent variable model given data by iterating two steps: generating model-based expectations of the latent variables, and maximizing the complete log-likelihood of the model given the data and latent expectations. Although the maximization step optimizes the complete log-likelihood, each iteration of EM is guaranteed to increase the data log-likelihood as well (Neal and Hinton, 1998).
EM is arguably the most widely-applied algorithm for fitting finite mixture models (McLachlan et al., 2019). As a form of latent variable exponential family, the expectation step for a finite mixture model reduces to computing average sufficient statistics, and the maximization step is a convex optimization problem (Wainwright and Jordan, 2008). In general, the average sufficient statistics, or mean parameters, correspond to (are dual to) the natural parameters of an exponential family, and where we denote natural parameters with θ, we denote their corresponding mean parameters with η.
Suppose we are given a dataset of neural spike-counts, and a CoM-based mixture model with natural parameters θN, , θK, and ΘNK (see Equation 13). The expectation step for this model reduces to computing the data-dependent mean parameters given by for all 0 < i ≤ d. The mean parameters are the averages of the sufficient statistic δk(k) under the distribution p(k | n(i)), and are what we use to complete the log-likelihood since we do not observe k.
Given, the maximization step of a CoM-based mixture thus reduces to maximizing the complete log-likelihood , where we substitute into the place of δ(k) in Equation 13, such that
This objective may be maximized in closed-form for a vanilla Poisson mixture (Karlis and Meligkotsidou, 2007), but this is not the case when the model has CoM-Poisson shape parameters or depends on the stimulus. Nevertheless, solving the resulting maximization step is still a convex optimization problem (Wainwright and Jordan, 2008), and may be approximately solved with gradient ascent. Doing so requires that we first compute the mean parameters ηN, , ηK, and HNK that are dual to θN, , θK, and ΘNK, respectively.
We compute the mean parameters by evaluating where ηK,k is the kth element of ηK, ηN,J is the Jth element of ηN, is the Jth element of , and ηNK,Jk is the Jth element of the kth column of HNK. Note as well that we truncate the series and to approximate μJk and . Given these mean parameters, we may then express the gradients of as where ⊗ is the outer product operator, and where the second term in each equation follows from the fact that the derivative of ψCK with respect to θN, , θK, or ΘNK yields the dual parameters ηN, , ηK, and HNK, respectively. By ascending the gradients of until convergence, we approximate a single iteration of the EM algorithm for a CoM-based mixture.
Finally, if our dataset includes stimuli x, and the parameters θN depend on the stimulus, then the gradients of the parameters of θN must also be computed. For a von Mises CPM where , the gradients are given by Where is the output of θN at x(i). Although in this paper we restrict our applications to Von Mises or discrete tuning curves for 1-dimensional stimuli, this formalism can be readily extended to the case where the baseline tuning curve parameters θN (x) are a generic nonlinear function of the stimulus, represented by a deep neural network. Then, the gradients of the parameters of θN can be computed through backpropagation, and is the error that must be backpropagated through the network to compute the gradients.
CPM initialization and training procedures
To fit a CPM to a dataset , we first initialize the CPM and then optimize its parameters with our previously described EM algorithm. Naturally, initialization depends on exactly which form of CPM we consider, but in general we first initialize the baseline parameters θN, then add the categorical parameters θK and mixture component parameters ΘNK. When training CoM-based CPMs we always first train a vanilla CPM, and so the initialization procedure remains the same for vanilla and CoM-based models.
To initialize a minimal, von Mises CPM with dN neurons, we first fit dN independent, von Mises-tuned neurons by maximizing the log-likelihood . This is a convex optimization problem and so can be easily solved by gradient ascent, in particular by following the gradients to convergence. For both discrete and maximal CPMs, where there are dX distinct stimuli, we initialize by computing the average rate vector at each stimulus-condition and creating a lookup table for these rate vectors. Formally, where xl is the lth stimulus value for 0 < l ≤ dX, we may express the lth rate vector as , where δ(xl, x(i)) is 1 when xl = x(i), and 0 otherwise. We then construct a lookup table for these rate vectors in exponential family form by setting , and by setting the lth row θNX,l of ΘNX to θNX,l = log λl+1 − log λ1.
In general we initialize the parameters θK by sampling the weights of a categorical distribution from a Dirichlet distribution with a constant concentration of 2, and converting the weights into the natural parameters of a categorical distribution θK. For discrete and maximal CPMs we initialize the modulations ΘNK by generating each element of ΘNK from a uniform distribution over the range [−0.0001, 0.0001]. For von Mises CPMs we initialize each row θNK,k of ΘNK as shifted sinusoidal functions of the preferred stimuli of the independent von Mises neurons. That is, given and ΘNX, we compute the preferred stimulus of the ith neuron given by , where θNX,i is the ith row of ΘNX. We then set the ith element θNK,k,i of θNK,k to . Initializing von Mises CPMs in this way ensures that each modulation has a unique peak as a function of preferred stimuli, which helps differentiate the modulations from each other, and in our experience improves training speed.
With regards to training, the expectation step in our EM algorithm may be computed directly, and so the only challenge is solving the maximization step. Although the optimal solution strategy depends on the details of the model and data in question, in the context of this paper we settled on a strategy that is sufficient for all simulations we perform. For each model we perform a total of dI = 500 EM iterations, and for each maximization step we take dS = 100 gradient ascent steps with the Adam gradient ascent algorithm (Kingma and Ba, 2014) with the default momentum parameters (see Kingma and Ba (2014)). We restart the Adam algorithm at each iteration of EM and gradually reduce the learning rate. Where ϵ+ = 0.002 and ϵ− = 0.0005 are the initial and final learning rates, we set the learning rate ϵt at EM iteration t to where we assume t starts at 0 and ends at dI − 1.
Because we must evaluate large numbers of truncated series when working with CoM-based CPMs, training times are typically one to two orders of magnitude greater. To minimize training time of CoM-based CPMs over the dI EM iterations, we therefore first train a vanilla CPM for 0.8dI iterations. We then equate the parameters θN, θK, and ΘNK of the vanilla CPM (see Equation 7) with a CoM-based CPM (see Equation 13) and set , which ensures that resulting CoM-based model has the same density function p(n, k | x) as the original vanilla model. We then train the CoM-based CPM for 0.2dI iterations. We found this strategy results in practically no performance loss, while greatly reducing training time.
CPM parameter selection for simulations
In the section Extended Poisson mixture models capture stimulus-dependent response statistics and the section CPMs facilitate accurate and efficient Bayesian decoding of neural responses we considered CoM-based, minimal CPMs with randomized parameters θN (x), , θK, and ΘNK, which for simplicity we refer to as models 1 and 2, respectively. We construct randomized CPMs piece by piece, in a similar fashion to our initialization procedure.
Firstly, where dN is the number of neurons, we tile their preferred stimuli ρi over the circle such that . We then generate the concentration, κi and gain γi of the ith neuron by sampling from normal distributions in log-space, such that log ,κi ~ N(−0.1, 0.2), and log γi ~ N(0.2, 0.1). Finally, for von Mises baseline tuning curves , we set each row θNX,i of ΘNX to θNX,i = (κi cos ρi, κi sin ρi), and each element of to , where ψX is the logarithm of the modified Bessel function of order 0, which is the log-partition function of the von Mises distribution.
We then set θK = 0, and generated each element θNK,i,k of the modulation matrix θNK in the same matter as the gains, such that θNK,i,k ~ N(0.2, 0.1). Finally, to generate random CoM-based parameters we generate each element of from a uniform distribution, such that .
Model 2 entails two more steps. Firstly, when sampling from larger populations of neurons, single modulations often dominate the model activity around certain stimulus values. To suppress this we consider the natural parameters of p(k | x) (see Equation 14), and compute the maximum value of these natural parameters over the range of stimuli . We then set each element θK,k of the parameters θK of the CPM to , where , which helps ensure that multiple modulations are active at any given x. Finally, since model 2 is a discrete CPM, we replace the von Mises baseline tuning curves with discrete baseline tuning curves, by evaluating at each of the dX valid stimulus-conditions, and assemble the resulting collection of natural parameters into a lookup table in the manner we described in our initialization procedures.
Decoding models
When constructing a Bayesian decoder for discrete stimuli, we first estimate the prior p(x) by computing the relative frequency of stimulus presentations in the training data. For the given encoding model, we then evaluate p(n | x) at each stimulus condition, and then compute the posterior p(x | n) ∝ p(n | x)p(x) by brute-force normalization of p(n | x)p(x). When training the encoding model used for our Bayesian encoders, we only trained them to maximize encoding performance as previously described, and not to maximize decoding performance.
We considered two decoding models, namely the linear network and the artificial neural network (ANN) with sigmoid activation functions. In both cases the input of the network was a neural response vector, and the output the natural parameters θX of a categorical distribution. The form of the linear network was θX (n) = θX + ΘXN · n, and is otherwise fully determined by the structure of the data. For the ANN on the other hand, we had to choose both the number of hidden layers, and the number of neurons per hidden layer. We cross-validated the performance of both 1 and 2 hidden layer models, over a range of sizes from 100 to 2000 neurons. We found the performance of the networks with 2 hidden layers generally exceeded that of those with 1 hidden layer, and that 700 and 600 hidden neurons was optimal for the awake and anaesthetized networks, respectively.
Given a dataset , we optimized the linear network and the ANN by maximizing via stochastic gradient ascent. We again used the Adam optimizer with default momentum parameters, and used a fixed learning rate of 0.0003, and randomly divided the dataset into minibatches of 500 data points. We also used early stopping, where for each fold of our 10-fold cross-validation simulation, we partitioned the dataset into 80% training data, 10% test data, and 10% validation data, and stopped the simulation when performance on the test data declined from epoch to epoch.
Experimental design
Throughout this paper we demonstrate our methods on two sets of parallel response recordings in macaque primary visual cortex (V1). The stimuli were drifting full contrast gratings at 9 distinct orientations spread evenly over the half-circle from 0° to 180° (2° diameter, 2 cycles per degree, 2.5 Hz drift rate). Stimuli were generated with custom software (EXPO by P. Lennie) and displayed on a cathode ray tube monitor (Hewlett Packard p1230; 1024 × 768 pixels, with ~ 40 cd/m2 mean luminance and 100 Hz frame rate) viewed at a distance of 110 cm (for anaesthetized dataset) or 60 cm (for awake dataset). Grating orientations were randomly interleaved, each presented for 70 ms (for anaesthetized dataset) or 150 ms (for awake dataset), separated by a uniform gray screen (blank stimulus) for the same duration.
For each electrode, we extracted waveform signals (sampled at 30 kHz) whenever the extracellular voltage exceeded a user defined threshold (typically 5x the root mean square signal on each channel). We then sorted waveforms manually using the Plexon Offline Sorter, and isolated both single and multi-unit clusters, here both referred to as neurons. We computed spike counts in a fixed window with length equal to the stimulus duration, shifted by 50 ms after stimulus onset. We excluded from further analyses all neurons that were not driven by any stimulus above baseline + 3std.
In the first dataset the monkey was awake, and there were dT = 3168 trials of the responses of 72 neurons; due to the presence of cross-talk between a small subset of electrodes, we removed all pairs of neurons in the dataset that exhibited correlations greater than 0.5, which left dN = 43 neurons in the dataset. We refer to this dataset as the awake V1 dataset. After familiarization with the restraining chair, headpost surgery, and postoperative recovery time (methods and protocols described in Festa et al. (2020)), the animal was trained to fixate in a 1.30 × 1.30 window. Eye position was monitored with a high-speed infrared camera (Eyelink, 1000 Hz). A second surgery was performed over V1 to implant a 96 channel microelectrode array into V1 (electrode length 1 mm). After postoperative recovery, the spatial receptive fields of the sampled neurons were mapped by presenting small patches of drifting full contrast gratings (0.50 diameter; 4 orientations, 1 cycle per degree, 3 Hz drift rate, 250 ms presentation) at 25 distinct positions spanning a 30 × 40 region of visual space. Subsequent stimuli were centered in the aggregate receptive field of the recorded units.
In the second dataset the monkey was anaesthetized and there were dT = 10, 800 trials of the responses of dN = 70 neurons; we refer to this dataset as the anaesthetized V1 dataset. The protocol and general methods employed for the anaesthetized experiment have been described previously (Smith and Kohn, 2008). In short, anaesthesia was induced with ketamine (10 mg/kg) and maintained during surgery with isoflurane (1.5–2.5% in 95% O2), switching to sufentanil (6–18 μg/kg per h, adjusted as needed) during recordings. Eye movements were reduced using vecuronium bromide (0.15 mg/kg per h). Temperature was maintained in the 36 –37 C0 range, and relevant vital signs (EEG, ECG, blood pressure, end-tidal PCO2, temperature, and lung pressure) were monitored continuously to ensure sufficient level of anaesthesia and well-being. A 10 × 10 multielectrode array (400 μm spacing, 1 mm length) was implanted into the upper layers of primary visual cortex, at a depth of 0.6-0.8 mm.
All procedures were approved by the Albert Einstein College of Medicine and followed the guidelines in the United States Public Health Service Guide for the Care and Use of Laboratory Animals.
Competing interests
The authors declare they have no competing interests.
Acknowledgments
We would like to thank all the members of the labs of Ruben Coen-Cagli and Adam Kohn for their regular feedback and support.