Abstract
Cortical responses to sensory stimuli are highly variable, and sensory cortex exhibits intricate spontaneous activity even without external sensory input. Cortical variability and spontaneous activity have been variously proposed to represent random noise, recall of prior experience, or encoding of ongoing behavioral and cognitive variables. Here, by recording over 10,000 neurons in mouse visual cortex, we show that spontaneous activity reliably encodes a high-dimensional latent state, which is partially related to the mouse’s ongoing behavior and is represented not just in visual cortex but across the forebrain. Sensory inputs do not interrupt this ongoing signal, but add onto it a representation of visual stimuli in orthogonal dimensions. Thus, visual cortical population activity, despite its apparently noisy structure, reliably encodes an orthogonal fusion of sensory and multidimensional behavioral information.
In the absence of sensory inputs, the brain produces structured patterns of activity, which can be as large or larger than sensory driven activity [1]. Ongoing activity exists even in primary sensory cortices, and have been hypothesized to reflect recapitulation of previous sensory experiences, or expectations of possible sensory events. This hypothesis is supported by studies that found similarities between sensory-driven and spontaneous firing events [2–4]. An alternative possibility is that ongoing activity could be related to behavioral and cognitive states. The firing of sensory cortical and sensory thalamic neurons correlates with behavioral variables such as locomotion, pupil diameter, and whisking [5–19]. Continued encoding of nonvisual variables when visual stimuli are present could at least in part explain the trial-to-trial variability in responses to repeated presentation of identical sensory stimuli [20].
The influence of trial-to-trial variability on stimulus encoding depends on its population-level structure. Variability that is independent between cells – such as the Poisson-like variability produced in balanced recurrent networks [21] – presents little impediment to information coding, as reliable signals can still be extracted as weighted sums over a large enough population. In contrast, correlated variability has consequences that depend on the form of the correlations. If correlated variability mimics the differences in responses to different stimuli, it can compromise stimulus encoding [22]. Conversely, variability in dimensions orthogonal to those encoding stimuli has no adverse impact on coding [23], and might instead reflect encoding of signals other than visual inputs.
Spontaneous cortical activity reliably encodes a high-dimensional latent signal
To distinguish between these possibilities, we characterized the structure of neural activity and sensory variability in mouse visual cortex. We simultaneously recorded from 11,262 ± 2,282 (mean ± s.d.) excitatory neurons, over nine sessions in six mice using 2-photon imaging of GCaMP6s in an 11-plane configuration [24] (Figure 1A,B, Movie S1). These neurons’ responses to classical grating stimuli revealed robust orientation tuning as expected in visual cortex (Figure S1).
We began by analyzing spontaneous activity in mice free to run on an air-floating ball. Six of nine recordings were performed in darkness, but we did not observe differences between these recordings (shown in red on all plots) and recordings with gray screen (yellow on all plots). Mice spontaneously performed behaviors such as running, whisking, sniffing, and other facial movements, which we monitored with an infrared camera.
Ongoing population activity in visual cortex was highly structured (Figure 1C-F). Correlations between neuron pairs were reliable (Figure S2), and their spread was larger than would be expected by chance (Figure 1C,D), suggesting structured activity [25]. Fluctuations in the first principal component (PC) occurred over a timescale of many seconds (Figure S3), and were coupled to variations in arousal levels, as indicated by running, whisking, and pupil area. These arousal-related variables were strongly correlated with each other (Figure S4A,B), and together accounted for approximately 50% of the variance of the first neural PC (Figure 1E, Figure S4C). Correlation with the first PC was positive or negative in approximately similar numbers of neurons (57% ± 6.7% SE positive), indicating that two large sub-populations of neurons alternate their activity (Figure 1F). The slowness of these fluctuations implies a different underlying phenomenon to previously-studied “up and down phases” [3, 12, 26–28], which alternate at a much faster timescale (100-300 ms instead of 10-20 s) and correlate with most neurons positively. Indeed, up/down phases could not even have been detected in our recordings, which scanned the cortex every 400 ms.
Spontaneous activity had a high-dimensional structure, more complex than would be predicted by a single factor such as arousal (Figure 1G). This structure could be visualized by sorting the raster diagram so that nearby neurons showed strong correlations (see also Figure S5). Position on this continuum bore little relation to actual distances in the imaged tissue (Figure S6).
Despite its noisy appearance, spontaneous population activity reliably encoded a high-dimensional latent signal (Figure 1H-K). To show this, we devised a method to identify dimensions of neural variance that are reliably determined by common underlying signals, termed Shared Variance Component Analysis (SVCA). The method begins by dividing the recorded neurons into two spatially segregated sets, and dividing the recorded timepoints into two equal halves (training and test; Figure 1H). The training timepoints are used to find the dimensions in each cell set’s activity that maximally covary with each other. These dimensions are termed Shared Variance Components (SVCs). Activity in the test timepoints is then projected onto each SVC (Figure 1I), and the correlation between projections from the two cell sets (Figure 1J) provides an estimate of the reliable variance in that SVC (see Methods and Appendix). The fraction of reliable variance in the first SVC was 97% (Figure 1I,J), implying that only 3% of the variance along this dimension reflected independent noise. The reliable variance fraction of successive SVCs decreased slowly, with the 50th SVC showing approximately 50%, and the 512th showing 9% (Figure 1K). Thus, visual cortical spontaneous activity encodes a multidimensional latent signal, and appropriately weighted sums of 10,000 neurons suffice to accurately extract ~100 dimensions of it.
The magnitude of reliable spontaneous variance was distributed across dimensions according to a power law of exponent 1.14 (Figure 1L). This value is larger than the power law exponents close to 1.0 seen for stimulus responses [29], but still indicates a high-dimensional signal. The first 128 SVCs together accounted for 86% ± 1% SE of the complete population’s reliable variance, and 67% ± 3% SE of the total variance in these 128 dimensions was reliable. Arousal variables accounted for 11% ± 1% SE of the total variance in these 128 components (16% of their reliable variance), and primarily correlated with the top SVCs (Figure 1M,N).
Ongoing neural activity encodes multidimensional behavioral information
Although arousal measures only accounted for a small fraction of the reliable variance of spontaneous population activity, it is possible that a larger fraction could be explained by higher-dimensional measures of ongoing behavior (Figure 2A-C, Movie S2). We extracted a 1,000-dimensional summary of the motor actions visible on the mouse’s face by applying principal component analysis to the spatial distribution of facial motion at each moment in time [30]. The first PC captured motion anywhere on the mouse’s face, and was strongly correlated with explicit arousal measures(Figure S4B), while higher PCs distinguished different types of facial motion. We predicted neuronal population activity from this behavioral signal using reduced rank regression: for any N, we found the N dimensions of the video signal predicting the largest fraction of the reliable spontaneous variance (Figure 2D).
This multidimensional behavior measure predicted approximately twice as much variance as the simple arousal variables (Figure 2D-J, Movie S3). To visualize how multidimensional behavior predicts ongoing population activity, we compared a raster representation of raw activity (vertically sorted as in Figure 1G) to the prediction based on multidimensional videography (Figure 2F, see Figure S5 for all recordings). To quantify the quality of prediction, and the dimensionality of the behavioral signal encoded in V1, we focused on the first 128 SVCs (accounting for 86% of the population’s reliable variance). The best one-dimensional predictor extracted from the facial motion movie captured the same amount of variance as the best one-dimensional combination of whisking, running, and pupil (Figure 2G). However, prediction quality continued to increase with up to 16 dimensions of videographic information (and beyond, in some recordings), suggesting that visual cortex encodes at least 16 dimensions of motor information. These dimensions together accounted for 21%± 1% SE of the total population variance (31% ± 3% of the reliable variance; Figure 2H), substantially more than the three-dimensional model of neural activity using running, pupil area and whisking (11% ± 1% SE of the total variance, 17% ± 1% SE of the reliable variance). Moreover, adding these three explicit predictors to the video signal increased the explained variance by less than 1% (Figure 2I), even though the running signal was not obtained from the video camera. A neuron’s predictability from behavior was not related to its cortical location (Figure S7). The timescale with which neural activity could be predicted from facial behavior was ~1 s (Figure 2J), reflecting the slow nature of these behavioral fluctuations.
Behaviorally-related activity is spread across the brain
The high-dimensional spontaneous activity patterns found in V1 were a reflection of activity patterns spread across the brain (Figure 3A-E). To show this, we performed large-scale electrophysiological recordings, using 8 Neuropixels probes [31] to simultaneously record from frontal, sensorimotor, visual and retrosplenial cortex, hippocampus, striatum, thalamus, and midbrain (Figure 3A,B). The mice were awake and free to rotate a wheel with their front paws. From recordings in three mice, we extracted 2,998, 2,768 and 1,958 units stable across ~1 hour of ongoing activity, and binned neural activity into 1.2 s bins, as for the imaging data.
Neurons were on average more strongly correlated with others in the same area, but substantial inter-area correlations also existed, suggesting non-localized patterns of neural activity (Figure 3C). All areas contained neurons positively and negatively correlated with the top facial motion PC, although thalamus and retrosplenial cortex contained predominantly arousal-preferring neurons (Figure 3D, p<10−8 two-sided Wilcoxon sign-rank test). Sorting the neurons by correlation again revealed a complex activity structure (Figure 3E). All brain areas contained a sampling of neurons from the entire continuum (Figure 3E, right), suggesting that a multidimensional structure of ongoing activity is distributed throughout the brain. This spontaneous activity spanned at least 128 dimensions, with 35% of the variance of individual neurons reliably predictable from population activity (Figure S8).
Similar to visual cortical activity, the activity of brainwide populations was partially predictable from facial videography (Figure 3F-H). Predictability of brain-wide activity again saturated around 16 behavioral dimensions, which predicted on average 16% of the total variance (42% of the estimated maximum possible) (Figure 3F). Other areas showed even stronger behavioral modulation than visual cortex, with neurons in thalamus predicted best (22% of total variance, 53% of estimated maximum). The timescale of videographic prediction was again broad: neural activity was best predicted from instantaneous behavior (17±24ms SE before behavior), but the predicted variance again decayed slowly over time lags of multiple seconds (Figure 3G). Taking advantage of the high temporal resolution of electrophysiological recordings, we reduced the analysis bin size from 1.2 seconds to 200 ms, revealing an additional sharp peak at short timescales across all areas (optimal timelag at 80±18 ms Figure 3H).
Stimulus-evoked and ongoing activity overlap along one dimension
We next asked how ongoing activity and behavioral information relates to sensory responses (Figure 4A-B). For this analysis, we interspersed blocks of visual stimulation with flashed natural images (presented 1 per second on average) with extended periods of spontaneous activity (gray screen), while imaging visual cortical population activity (Figure 4A). During stimulus presentation, the mice continued to exhibit the same behaviors as in darkness, resulting in a similar distribution of facial motion components (Figure 4B).
Representations of sensory and behavioral information were mixed together in the same cell population. There were not separate sets of neurons encoding stimuli and behavioral variables: the fraction of each neuron’s variance explained by stimuli and by behavior were only slightly negatively correlated (Figure S9; r = -0.18, p<0.01 Spear-man’s rank correlation), and neurons with similar stimulus responses did not have more similar behavioral correlates (Figure S9; r = -0.005, p > 0.05).
Nevertheless, the subspaces encoding sensory and behavior information overlapped in only one dimension (Figure 4C-E). The space encoding behavioral variables contained 11% of the total stimulus-related variance, 96% of which was contained in a single dimension (Figure 4C) with largely positive weights onto all neurons (85% positive weights, Figure 4D). Similarly, the space of ongoing activity, defined by the top 128 principal components of spontaneous firing, contained 23% of the total stimulus-related variance, 86% of which was contained in one dimension (85% positive weights). Thus, overlap in the spaces encoding sensory and behavioral variables arises only because both can change the mean firing rate of the population: the precise patterns of increases and decreases about this change in mean were essentially orthogonal (Figure 4E).
To visualize how the V1 population integrated sensory and behavior-related activity, we examined the projection of this activity onto three orthogonal subspaces: a multidimensional subspace encoding only sensory information (stim-only); a multidimensional subspace encoding only behavioral information (behav-only); and the one-dimensional subspace encoding both (stim-behav shared dimension) (Figure 4F-G; Figure S10). During gray-screen periods there was no activity in the stim-only subspace, but when the stimuli appeared this subspace became very active. Conversely, activity in the behav-only subspace was present prior to stimulus presentation, and continued unchanged when the stimulus appeared. The one-dimensional shared subspace showed an intermediate pattern: activity in this subspace was weak prior to stimulus onset, and increased when stimuli were presented. Similar results were seen for the spont-only and stimspont spaces (Figure 4F, lower panels). Across all experiments, variance in the stim-only subspace was 119 ± 81 SE times larger during stimulus presentation than during spontaneous epochs (Figure 4G), while activity in the shared subspace was 19 ± 12 SE times larger; activity in the face-only and spont-only subspaces was only modestly increased by sensory stimulation (1.4 ± 0.13 SE and 1.7 ± 0.2 SE times larger, respectively).
To visualize how stimuli affected activity in these subspaces, we plotted population responses to multiple repeats of two example stimuli (Figure 4H-K). When projected into the stim-only space, the resulting clouds were tightly defined with no overlap (Figure 4H), but in the behav-only space, responses to the two stimuli were directly superimposed (Figure 4I). Variability within the stimulus subspace consisted of changes in the length of the projected activity vectors between trials, resulting in narrowly elongated clouds of points, consistent with previous reports of multiplicative variability in stimulus responses [32–35]. A model in which stimulus responses are multiplied by a trial-dependent factor accurately captured the data, accounting for 89% ± 0.1% SE of the variance in the stimulus subspace (Figure 4J). Furthermore, the multiplicative gain on each trial could be predicted from facial motion energy (r = 0.61 ± 0.02 SE, cross-validated), and closely matched activity in the shared subspace (r = 0.73 ± 0.06 SE, cross-validated; Figure 4K). Although ongoing activity in the behav-only space and visual responses in the stim-only subspace added independently, we did not observe additive variability within the stim-only space itself: an “affine” model also including an additive term did not significantly increase explained variance over the multiplicative model (p > 0.05, Wilcoxon rank-sum test).
Discussion
Ongoing population activity in visual cortex reliably encoded a latent signal of at least 100 linear dimensions, and possibly many more. The largest dimension correlated with arousal and modulated about half of the neurons positively and half negatively. At least 16 further dimensions were related to behaviors visible by facial videography, which were also encoded across the forebrain. The dimensions encoding motor variables overlapped with those encoding visual stimuli along only one dimension, which coherently increased or decreased the activity of the entire population. Activity in all other behavior-related dimensions continued unperturbed regardless of sensory stimulation. Trial-to-trial variability of sensory responses comprised additive ongoing activity in the behavior subspace, and arousal-dependent multiplicative modulation in the stimulus subspace, resolving apparently conflicting findings concerning the additive or multiplicative nature of cortical variability [20, 32–35].
Our data are consistent with previous reports describing low-dimensional correlates of locomotion and arousal in visual cortex [5, 7–14], but suggest these results were glimpses of a much larger set of behavioral and cognitive variables encoded by ongoing activity patterns. We found that 16 dimensions of facial motor activity can predict 31% of the reliable spontaneous variance. The remaining dimensions and variance might in part reflect motor activity not visible on the face or only decodable by more advanced methods [36–39], or they might reflect internal cognitive variables such as motivational drives.
The fact that ongoing and visually-evoked activity overlap in only one dimension at first appears to contradict previous reports showing similarity of sensory responses to ongoing activity [2–4]. We suggest three possible explanations for this apparent discrepancy. First, our experiments looked at a slower timescale than most previous studies, which binned the data at 100 ms [3], or even 2 ms [4]. Second, even a single dimension of common rate fluctuation is sufficient for some previously-applied statistical methods to report similar population activity [40]. Third, we found that encoding of non-sensory variables continued throughout stimulus presentation. Thus, similar firing patterns during stimulation and ongoing activity need not imply recapitulation of sensory events, just that cortex continues to encode nonsensory variables during sensory stimulation.
The brainwide representation of behavioral variables suggests that information encoded nearly anywhere in the forebrain is combined with behavioral state variables into a mixed representation. We found that these multidimensional signals are present both during ongoing activity and during passive viewing of a stimulus. Recent evidence indicates that they may also be present during a decision-making task [41]. What benefit could this ubiquitous mixing of sensory and motor information provide? The most appropriate behavior for an animal to perform at any moment depends on the combination of available sensory data, ongoing motor actions, and purely internal variables such as motivational drives. Integration of sensory inputs with motor actions must therefore occur somewhere in the nervous system. Our data indicate that it happens as early as primary sensory cortex. This is consistent with neuroanatomy: primary sensory cortex receives innervation both from neuromodulatory systems carrying state information, and from higher-order cortices which can encode fine-grained behavioral variables [6]. This and other examples of pervasive whole-brain connectivity [42–46] may coordinate the brain-wide encoding of behavioral variables we have reported here.
Author Contributions
Conceptualization, C.S., M.P., N.S., M.C. and K.D.H.; Methodology, C.S. and M.P.; Software, C.S. and M.P.; Investigation, C.S., M.P., N.S. and C.B.R.; Writing, C.S., M.P., N.S., M.C. and K.D.H; Resources, M.C. and K.D.H. Funding acquisition, M.C. and K.D.H.
Declaration of interests
The authors declare no competing interests.
Movie S1. Spontaneous neural activity of 10,000+ neurons in visual cortex of awake mice. Two-photon calcium imaging of 11 planes spaced 35 μm apart. Movie speed is 10x real-time.
Movie S2. Multi-dimensional spontaneous behaviors. Movie speed is 5x real-time.
Movie S3. Spontaneous behaviors are correlated with spontaneous neural activity. Video of mouse face recorded simultaneously with neural activity. Movie speed is 10x real-time.
Acknowledgements
We thank Michael Krumin for assistance with the two-photon microscopes, and Andy Peters for comments on the manuscript.
This research was funded by Wellcome Trust Investigator grants (095668, 095669, 108726, and 205093) and by a grant from the Simons Foundation (SCGB 325512). CS was funded by a four-year Gatsby Foundation PhD studentship. MC holds the GlaxoSmithKline / Fight for Sight Chair in Visual Neuroscience. CS and MP are now funded by HHMI Janelia. NS was supported by postdoctoral fellowships from the Human Frontier Sciences Program and the Marie Curie Action of the EU (656528).