## Abstract

The brain has no direct access to physical stimuli, but only to the spiking activity evoked in sensory organs. It is unclear how the brain can structure its representation of the world based on differences between those noisy, correlated responses alone. Here we show how to build a distance map of responses from the structure of the population activity of retinal ganglion cells, allowing for the accurate discrimination of distinct visual stimuli from the retinal response. We introduce the Temporal Restricted Boltzmann Machine to learn the spatiotemporal structure of the population activity, and use this model to define a distance between spike trains. We show that this metric outperforms existing neural distances at discriminating pairs of stimuli that are barely distinguishable. The proposed method provides a generic and biologically plausible way to learn to associate similar stimuli based on their spiking responses, without any other knowledge of these stimuli.

A major challenge in neuroscience is to understand how the brain processes sensory stimuli. In particular, the brain must learn to group some stimuli in the same category, and to discriminate others. Strikingly, this feat is achieved while the brain has only access to the noisy responses evoked in sensory organs, but never to the stimulus itself. For example, the brain only receives the retinal responses to visual stimuli, and is able to associate together responses corresponding to the same stimulus, while teasing apart the ones coming from distinguishable stimuli. How nervous systems can achieve such discrimination is still unclear. One strategy to solve this problem could be to learn either a decoding model to reconstruct the stimulus from the neural responses [1, 2], or an encoding model and invert it to find stimuli that can be distinguished [3]. However, in both cases, this requires to have access to a lot of pairs of stimuli and evoked responses. Clearly, the brain is not guaranteed to have access to such data, and may only access the neural response without knowing the corresponding stimulus.

Neural metrics, which define a distance between pairs of spike trains, have been proposed to solve this issue. In general, spike trains evoked by the same stimulus should be close by, while spike trains corresponding to very different stimuli should be far away. Using a given metric, one can associate together responses evoked by similar stimuli, without any information about the stimuli themselves [4, 5]. The quality of this classification relies on the metric being well adapted to the task at hand, and different metrics are not expected to perform equally well.

Multiple metrics based on different features of the neural response have been proposed, mostly for single cells [6–11], and exceptionally for populations [12]. These metrics do not use information about the correlative structure of the population response, and often require to tune parameters to optimize performance, which requires external knowledge of the stimulus. In addition, a precise quantification of the performance of these different metrics at discriminating barely distinguishable stimuli is lacking.

Here we present an approach to learn a spike train metric with high discrimination capacity from the statistical structure of the population activity itself. We applied the method to the retina, a sensory system characterized by noisy, non-linear [13], and correlated [14, 15] responses. We first introduce a statistical model of retinal responses, the Temporal Restricted Boltzmann Machine, which allows us to learn an accurate description of spatio-temporal correlations in a population of 60 ganglion cells of the rat retina, stimulated by a randomly moving bar. We then use this model to derive a metric on neural responses. Using closed-loop experiments, where stimuli are tuned to be hardly distinguishable from each other, we show that this neural metric outperforms classical metrics at stimulus discrimination tasks. This high discrimination capacity is achieved despite the neural metric being trained with no information about the stimulus. We therefore suggest a general and biologically realistic method for the brain to learn to efficiently discriminate stimuli solely based on the output of sensory organs.

## RESULTS

### >Modeling synchronous population activity with Restricted Boltzmann Machines

We analyzed previously published *ex vivo* recordings from rat retinal ganglion cells [16]. A population of 60 cells was stimulated with a moving bar and recorded with a multielectrode array (Fig. 1). Responses were binarized in 20 ms time bins, with value σ_{it} = 1 if neuron *i* spiked during in given time bin *t*, and 0 otherwise (Fig. 1). We first aimed to describe the collective statistics of spikes and silences in the retinal population, with no regard for the sequence of stimuli that evoked them.

We modeled synchronous correlations between neurons using Restricted Boltzmann Machines (RBMs) [17, 18], which have previously been applied to retinal [19, 20] and cortical [21] populations. They give the probability of same-time spikewords (*σ*_{i}) = (*σ*_{it})_{i} at any *t* as:

RBMs do not have direct interactions between neurons. Rather, their correlations are explained by interactions with binary latent variables, *h*_{j}, called hidden units (Fig. 2A). When a hidden unit takes value 1, it induces collective changes in the excitability of sub-populations of cells. Although it is tempting to think of hidden units as non-visible neurons, they are only effective variables and usually do not correspond to actual neurons; hidden units can in fact reflect multiple causes of correlations, such as direct input from neighboring cells, or common input from intermediate layers and the stimulus. Their number can be varied: the more hidden units, the more complex structures can be reproduced by the model, but the more parameters need to be estimated.

We learned an RBM with 20 hidden units to model the responses of the retinal population responding to a randomly moving bar. The model was inferred on a training set (80% of responses) using persistent contrastive divergence (Materials and Methods), and its predictions compared to a testing set (20% remaining responses). The RBM predicted well each neuron’s firing rate (Fig. 2B) as well as correlations between pairs of neurons (Fig. 2D). In addition, the RBM predicted higher-order correlations accurately, such as the distribution of the total number of spikes in the population (Fig. 2C). By contrast, a model of independent neurons (zero hidden units) underestimated the probability of events with few or many spikes by an order of magnitude. The model performance, measured by either the fraction of variance explained of pairwise correlations (Fig. 2E), or by the model log-likelihood (Fig. 2F), quickly saturated with the number of hidden units, with 15 units already providing near optimal performance.

### Temporal Restricted Boltzmann Machines for population spike trains

The RBM performs well at modeling neural responses within 20 ms time bins, but correlations between neurons often span longer time scales. To evaluate the importance of these longer term correlations, we plotted the distribution of the number of spikes in the population in a 100 ms time window (using the testing set), and compared it to the prediction from the RBM, where the response of the population was generated in each of the five 20-ms bin independently (Fig. 3C). Although the RBM performed better than a model of independent neurons, it still underestimated the probability of large numbers of spikes by an order of magnitude, indicating that correlations over longer scales than 20 ms play an important role in shaping the collective response statistics.

To account for these temporal correlations, we introduced the Temporal Restricted Boltzmann Machine (TRBM). This model generalizes the RBM by allowing for interactions between neurons and hidden units across different time bins (Fig. 3A, Materials and Methods):

Because we want to describe the stationary distribution of spike trains regardless of the stimulus, absolute time is irrelevant, and the model is invariant to time translations: connections between a hidden unit and a neuron only depend on the relative delay *t*′ − *t* between them. This property is similar to convolutional networks used in image processing, but here in time instead of space.

We trained a TRBM with 10 hidden units per time bin, each connected to neurons across 5 consecutive time bins, on the same training set as before using persistent contrastive divergence (Materials and Methods), and compared predictions to the testing set. Like the RBM, the TRBM could predict individual neuron firing rates (Fig. 3B) and synchronous pairwise correlations (Fig. 3D). In addition, the TRBM could also predict temporal correlations ignored by the RBM. In particular it reproduced accurately the distribution of the total number of spikes in a 100 ms time window, that the RBM did not (Fig. 3C). We also tested if the TRBM could predict correlations between the spiking activity of pairs of neurons in two time bins separated by a given delay. To do so, we computed the total variance of pairwise correlations for a each delay, and estimated the fraction of it that could be explained by the TRBM (Fig. 3E, Materials and Methods). Even though direct connections between neurons and hidden units were limited to 80 ms, the TRBM could explain a substantial amount of correlations even for large delays, up to 150 ms where correlations all but vanish.

Similarly to the RBM, we found that increasing the number of hidden units only marginally improved performance (as measured by the fraction of explained variance of pairwise correlations) beyond 10 units per time bin (Fig. 3F). We also varied the maximum connection delay between neurons and hidden units from 20 ms to 120 ms. Performance quickly saturated at a connection delay of around 60 ms (Fig. 3G). In the following we will consider a TRBM with 10 hidden units and connection delay of 80 ms, unless mentioned otherwise.

### A neural metric based on response statistics

The hidden units of the TRBM can be considered as a way to compress the variability present in the neural activity, and extract its most relevant dimensions. We asked whether these hidden units could be used to define a neural metric that would follow the structure of the population code, allowing for efficient discrimination and classification properties.

To this end, we designed neural metrics derived from the RBM and TRBM based on the difference between the hidden unit states. Take two responses ** σ** = (

*σ*

_{i}) and of the retina, and define

**Δ**= (Δ

*h**h*

_{j}) as the difference of mean value of the hidden units conditioned on the two responses, Δ

*h*

_{j}= 〈

*h*

_{j}〉

_{σ}– 〈

*h*

_{j}〉

_{σ′}(Materials and Methods). Then the RBM metric is defined as: where

**= 〈**

*C*

*σσ*^{T}〉 – 〈

**〉 〈**

*σ*

*σ*^{T}〉 is the covariance matrix of the response, and

**= (**

*W**W*

_{ji}) is the matrix of couplings between neurons and hidden units. This definition can readily be generalized to the TRBM by adding time indices (Materials and Methods). Note that this metric differs from the Euclidian distance in the space of hidden units,

**Δ**

*h*^{T}

**Δ**

*: it has a nontrivial kernel*

**h**

*WCW*^{T}, which modulates the contribution of each hidden unit by its impact on neural activity. We will see later that this kernel improves discrimination capabilities. Note that this metric was defined without any information about the stimulus, and solely from the knowledge of the activity. We next aimed to test how well this metric can discriminate pairs of stimuli.

### Distinguishing close stimuli

To evaluate the capacity of a neural metric to finely resolve stimuli based on the sensory response, we introduce a measure a discriminability between the responses to two distinct stimuli based on neural metrics.

The response to a given stimulus is intrinsically noisy. Two repetitions of the same stimulus (let us call it reference stimulus) will give rise to two distinct responses *R*_{ref} and . The response *R*_{pert} to a perturbation of the reference stimulus may thus be hard to tease apart from another response to the reference stimulus, because of this noise (Fig. 5A). Given a neural metric *d*(*R*, *R*′), it is natural to define the discriminability of a perturbation as the probability for the response *R*_{pert} to be further apart from a response to the reference, , than would two responses to the reference, *R*_{ref} and , from each other:

If a perturbation is perfectly discriminable (Fig. 5A, left), distances between reference and perturbation are well separated from distances within responses to the reference, and the discriminability will approach 1. Conversely, for perturbations too small to be discriminated, the two distributions greatly overlap (Fig. 5A, right), and the discriminability is close to 0.*5* corresponding to chance.

To finely assess the capacity of neural metrics to perform discrimination tasks, we need to study perturbations that lie between these two extremes, where discrimination is neither easy nor impossible. To find this soft spot, we performed closed-loop experiments where at each step the discriminability of a perturbation was analyzed in order to generate the perturbation at the next step (Fig. 4B, see [16] for more details). We first recorded multiple responses to a reference stimulus, a 0.*9* s snippet of bar trajectory described earlier (Fig. S1 A-B). We then recorded responses to many perturbations of this stimulus (Fig. 4C). For a given “shape” of the perturbation (i.e. normalized difference of bar position between reference and perturbation as a function of time, Fig. S1 C), we adapted the perturbation size online, and searched for the smallest perturbations that were still discriminable (Materials and Methods). If a perturbation had high dis-criminability (as defined by a linear discrimination task on the thresholded values of the raw multi-electrode array output, independently of any metric, see Materials and Methods), at the next step we tested a perturbation with smaller amplitude. Conversely, if a perturbation had low discriminability, we then tested a larger perturbation. Perturbations lasted 320 ms, and responses were analyzed over 300 ms with a delay (Fig. 4D).

Thanks to this method, we could explore the space of possible perturbations efficiently, exploring multiple directions (shapes) of the perturbation space simultaneously, and obtained a range of responses to pairs of stimuli that are challenging but not impossible to discriminate. This method allowed us to benchmark different metrics.

### TRBM metric outperforms other neural metrics at fine discrimination tasks

We measured the discriminability (Eq. 4) of a perturbation at different amplitudes, using the RBM and TRBM metrics (Fig. 5A, Materials and Methods). As expected, the discriminability increased with the perturbation amplitude, with small perturbations being hardly discriminable from the reference stimulus (discriminabil-ity close to 0.5), and large perturbations almost perfectly discriminable (discriminability close to 1). Since this metric is based on the hidden states, it means that hidden states are informative about the stimulus. The much better performance of the TRBM, especially for small and medium perturbations, emphasizes the importance of temporal correlations in shaping the metric. For comparison, we computed the discriminability of the same perturbation for the Victor-Purpura metric [6] (Materials and Methods), one of the first proposed neural metrics which has often been used in the literature to estimate the sensitivity of neural systems [22–24]. This metric depends on a time scale parameter, which we optimized to maximize the mean discriminability of all recorded responses. Even with this optimization, the Viktor-Purpura metric discriminated perturbations less well than either the RBM or TRBM metrics, whose parameters were not optimized, across all perturbation amplitudes.

To see if this better performance of our TRBM metrics held for other stimuli, we compared the discrimination capacity of the RBM and TRBM metrics with the Victor-Purpura metric, for 2 different reference stimuli and 16 perturbation shapes for each (Fig. S1). For each reference stimulus and perturbation shape, we separated responses in batches of low, medium and high discriminability, based on a linear discrimination task independent of any metric (Materials and Methods). We computed the mean discriminability of each response batch, for the RBM, TRBM, and Victor-Purpura metrics (Fig. 5B and C). While responses in the low discriminability batch were poorly separated by all three metrics, a large majority of responses with medium and high discriminability had larger discriminability for the RBM metric (Fig. 5B), and even larger for the TRBM metric (Fig. 5C), confirming the importance of temporal correlations.

We then compared the RBM and TRBM metrics to other neural metrics from the literature: van Rossum, angular, inter-spike interval (ISI), nearest neighbor, event synchronization, spike synchronization, and SPIKE metrics (definitions in Materials and Methods), as well as the simple Hamming distance on the binarized responses. Metrics with free parameters were optimized to maximize their mean discriminability. For each metric, we computed the mean discriminability in each batch (low, medium or high discriminability) across all reference stimuli and perturbation shapes (Fig. 5C-E). Responses from the low-discriminability batch were hard to distinguish, and only five metrics did significantly better than chance (*p* < 0.05 for unpaired *t*-test, Fig. 5C): RBM, spike synchronization, SPIKE, Angular and TRBM metrics. The TRBM metric discriminated responses the best, and was significantly better than the second best, the SPIKE metric (*p* = 0.014, paired *t*-test). For the medium and high discriminability batches, the RBM and TRBM metrics greatly outperformed all other metrics. Strikingly, in the medium discriminability group, the improvement of discriminability above chance level was 30% higher for the RBM metric, and 94% for the TRBM metric, than for the Angular metric, the most discriminating metric from the literature.

This performance was little affected by the number of hidden units in the RBM and TRBM. The mean dis-criminability increases and eventually saturates with the number of hidden units (Fig. S2), indicating that the metric was not sensitive to that precise number, provided that it is large enough. By contrast, the TRBM-based metric using the *Euclidian* distance between the mean values of the hidden units degraded quickly with the number of units (dashed lines in Fig. S2). This worse performance may be explained by the fact that some hidden units have little or redundant impact on the activity, but are counted with equal weight in the Euclidian distance. This stresses the importance of accounting for the impact of hidden units on the activity through a distance kernel as in Eq. 3.

Finally, we checked that our conclusions were not affected by the choice of bin size. We repeated all procedures for RBM and TRBM with time bins of size 5, 10 and 40 ms, and obtained consistent results (data not shown).

## DISCUSSION

We have designed a novel and general method to build a metric from the neural responses of sensory organs, which outperforms all previously defined metrics when trying to discriminate stimuli. This metric called, the TRBM metric, is based on a statistical model of the activity. Importantly, that model is trained in an *unsupervised* way, meaning that no knowledge of the stimulus was required in the learning procedure. Previous work has considered the possibility of constructing ‘semantic’ metrics based on the relationship between the retinal response and the stimuli that evoked them, and proposed a clustering scheme based on that metric [3]. Although such a metric clearly follows the structure of the population code, as does our own, it presents an extreme case of supervised learning, as building this metric in practice requires to know the probability of response triggered by all possible stimuli, which the brain cannot do in practice. By contrast, the TRBM metric discriminates stimuli well, but rather than being learned from the stimulus-response dictionary, it naturally emerges from learning the structure of the retinal code, which can be done with a reasonable amount of data and without any information about the stimulus. This way of constructing a metric suggests a realistic strategy for the brain to learn to discriminate stimuli. Note that many neural metrics require to tune parameter values to maximize performance—a supervised process as it uses knowledge of the stimulus in evaluating discrimination *a posteriori*. Strikingly, even after this optimization, the TRBM metric outperforms all metrics we found in the literature.

Although we have motivated introducing the TRBM for defining a metric, statistical models of population activity deserve attention in their own right. In this regard, the TRBM provides an alternative to existing approaches that is both accurate and tractable. The spiking responses of retinal ganglion cells at a given time are strongly correlated [15, 25], and various strategies have been proposed to model their collective, synchronous (same time-bin) activity. Central to this effort are models based on the principle of maximum entropy [15, 26–28], which allow for an explicit mapping onto models of Ising spins from statistical mechanics, also known as Boltzmann machines [29]. However these models are often hard to learn in practice, and they need additional terms [30, 31] or non-linearities [32] to explain higher-order statistics such the distribution of total number of spikes. It also is unclear how to exploit their structure to derive a metric.

As an alternative to Ising models, RBMs were applied to the correlated activity in cortical micro-columns [21] and in the retina [19, 20]. Our results confirm their ability to describe the synchronous collective activity in the retina, including pairwise correlations and the distribution of total numbers of spikes. Previous work also showed that the hidden units of a variant of the RBM conveyed information about the stimulus, although this was only made possible by the small number of used stimuli [33]. All these models ignore correlations between spikes in different time bins, which play an important role as we have shown here. Maximum entropy models were generalized to account for correlations across time, but they were either practically intractable for large populations [34, 35], or only focused on the total number of spikes [36]. The TRBM can reproduce pairwise and higher-order correlations with high accuracy across different time bins, and this with a reasonable amount of parameters and with relative computational ease. We therefore expect the TRBM to be useful in describing the stimulus-independent activity of neural populations in a variety of contexts [37–40].

Also using latent variables, a Hidden Markov Model (HMM) was proposed to describe retinal activity, in which the response is controlled by a hidden categorical variable following a Markov chain [41]. However, to reflect the diversity of responses, the number of categories encoded by the hidden variable should grow exponentially with the population size, which is impractical computationally and biologically. By contrast, the number of configurations of the hidden state in the TRBM grows exponentially with the number of hidden units, making easy its application to large populations. Continuous latent variables have also been proposed to account for neural correlations in cortical networks, such as in the linear dynamical system [42]. It could be interesting to apply such models in the retina. However, complex computational techniques are needed to infer them, while the TRBM is relatively simple to learn.

Our work presents an example of unsupervised learning (inferring the TRBM) which proves predictive in a supervised task (stimulus discrimination). This is reminiscent of the technique of unsupervised ‘pre-training’ [43], which is used in machine learning when only a few examples are available to improve model performance. The link to machine learning suggests to consider “deep” extensions of the RBM, with several layers of hidden variables [44], from which more general metrics could be derived. It has been shown that deep (artificial) neural networks achieve higher discrimination power than RBMs when dealing with complex stimuli such as natural scenes, and such complex architectures could lead to better metrics in our case as well.

The TRBM was trained on responses to the random motion of a bar. One may wonder how well the TRBM and its derived metric would perform when confronted with stimuli from different statistics (e.g. random checkerboard, natural movies). One possibility is that the brain constantly re-learns the metric depending on the visual stimulus, or alternatively learns a universal metric that performs well across a wide random of stimulus conditions. To address this question, we would need not only to display several different stimulus ensembles, but also to design an experimental procedure for comparing responses to pairs of close enough stimuli in that ensemble. Studying this question is an interesting avenue for future research.

While neural metrics may not be explicitly estimated by the brain, our TRBM metrics have a natural biological implementation that suggests how a downstream population could discriminate responses to different stimuli. Hidden units could be implemented by a population of downstream neurons, with a simple response function: a weighted sum followed by a nonlinearity (eq. 10). This is reminiscent of a neuron summing responses from upstream cells, weighted by synapses’ strengths, with delays to account for time lags. Indeed, it was shown that networks of spiking neurons can learn their synaptic weights to approximate Restricted Boltzmann Machines [45]. One can simplify our TRBM metric by linearizing the dependence of the hidden units as a function of activity (see Materials and Methods). Doing so leads to a metric that readily generalizes to continuous times, where the binning of time disappears. The metric is then simply given by a sum over pairs of spikes, with coefficients depending on the identity of the spiking neurons and the delay between them. We showed that this simplified “continuous” TRBM performs almost as well as the full TRBM metric (Fig. S3). The continuous TRBM metric could be implemented by simple summation of spikes with time delays.

In summary, the TRBM provides insights into biologically possible representations of the stimulus with high discrimination capabilities, without the need for any supervised training. None of the properties of the TRBM and its derived metric are expected to be specific to the retina, and our method could be readily applied to other sensory neural circuits.

## MATERIALS AND METHODS

### A Electrophysiology

We analyzed previously published *ex vivo* recordings of retinal ganglion cells from a male Long Evans rat [16]. In brief, the animal was killed according to institutional animal care standards. The retina was extracted from the animal, maintained in an oxygenated Ames solution, and recorded on the ganglion cell side with a 252-electrode array. Spike sorting was performed with custom software [46], and *N* = 60 neurons were selected for the stability of their spike waveforms and firing rates, the lack of refractory period violation, and the consistency of their responses to repeated stimuli.

### B Stimulus

The stimulus was a movie of a white bar on a dark background projected at 50 Hz with a digital micromirror device. The bar had intensity 7.6 × 10^{11} photons.cm^{−2}.s^{−1}, and 115 μm width.

The bar trajectory was composed of two interleaved parts. The first part was composed of multiple 0.9 s-long snipets of random motion (a Brownian motion with a restoring force). The second part was composed of 391 repetitions of two trajectories lasting 0.9 s each, which we call reference trajectories (Fig. S1 A-B). Both parts were generated with the same random motion. The two parts were alternated for a total of 15331 s.

We also presented perturbations of the reference trajectories. Perturbations were small changes affecting each reference trajectory in its middle portion, between 280 and 600 ms. Perturbations varied both in shape and in amplitude: we used 16 different perturbation shapes (Fig. S1 C), each presented at different amplitudes (Fig. 4B). The amplitude of perturbations was adapted online: large enough so they could be discriminated from reference trajectories, but small enough so they would not be discriminated perfectly with any metric.

Formally, if we denote a perturbation by its discretized time series with time step Δ_{t} = 20 ms, *S* = (*S*_{1},…,*S*_{16}), it can be decomposed as *S* = *A* × *P*, where is the amplitude, and *P* = *S*/*A* the shape. More details can be found in [16].

For each perturbation, the response was considered from 160 ms after the start of the perturbation until 140 ms after its end, so that responses lasted 300 ms (Fig. 4C).

### C Models of response distributions

In order to model neural correlations, the response of *N* neurons was binarized into time bins of size Δ_{t} = 20 ms: *σ*_{i} = 1 if neuron *i* spiked, and 0 otherwise (Fig. 1).

### 1 Restricted Boltzmann Machine

We used Restricted Boltzmann Machines (RBM) to model the probability of responses within a single time bin. RBM are statistical models with no direct interactions between the *N* neurons. Instead, the neurons interact with *M* binary latent variables, termed hidden units. There is no direct interactions between hidden units either. If we call *h*_{j} the hidden unit *j*, the joint probability between neurons and hidden units takes the form *P*(** σ,h**) ∝

*e*

^{−E}with: or in matrix form: where the bold font stands for vectors and matrices, and

*h*^{T}stands for the transpose of

**. The hidden variables do not necessarily correspond to existing entities interacting with neurons. Instead, they are effective variables that are used to capture correlations between neurons.**

*h*This model is convenient to simulate because neurons (resp. hidden units) conditioned by hidden units (resp. neurons) are independent. Namely, given a state of hidden units ** h**, neurons are independent and:
where

*f*(

*x*) = 1/(1 +

*e*

^{−x}is the sigmoid function. The conditional probability of hidden units given the neurons can be computed with a similar formula.

We learned the RBM on 80% of responses to the random trajectories of the bar, with *M* = 20 hidden units. We inferred the model by maximizing the likelihood using persistent contrastive divergence [47] with 200 epochs and minibatches of size 10. We used the momentum method [48], which is known to accelerate learning, with a momentum coefficient of 0.9. This method updates parameters in a direction proportional to the sum of the likelihood gradient and the parameter update at the previous step. For regularization, we used a weight decay parameter of 10^{−5} [49], meaning that the objective function maximized during learning was the sum of the loglikelihood and of the Euclidean norm of coupling parameters ** W**, weighted by a factor 10

^{−5}. For computation of model statistics, we simulated the model using block Gibbs Sampling [48]. Final model statistics (Fig. 2) were computed with simulations with 300 steps on responses spanning 2 × 10

^{6}time bins.

### 2 Temporal Restricted Boltzmann Machine

In order to model interactions between neurons across multiple time bins, we use a more complex RBM, allowing connections between neurons and hidden units in different time bins. We call it the Temporal Restricted Boltzmann Machine (TRBM). In order to limit the number of parameters, this model is convolutional [50], meaning that the interaction between a neuron and a hidden unit does not depend on their absolute time, but only on the delay *d* between them. We define *σ*_{ik} as the response of neuron *i* in time bin *k*. The probability of neurons and hidden units during *K* time bins takes the form *P*(** σ,h**) ∝

*e*

^{−E}with: where the time span of interactions between hidden and visible units is

*D*. It can be noted that in the case

*D*= 1, the TRBM consists of an independent RBM in each time bin. The TRBM can also be written in matrix form: where

*W*_{d}are matrices of size

*M*×

*N*.

The parameters can be learned independently of the length of responses considered, and can be used to model responses of different lengths. Visible (resp. hidden) units at the time boundaries depend on hidden (resp.visible) units outside boundaries. However, given a finitesize response we can still compute exactly the probability of some hidden units using:

Therefore, during learning we only consider hidden units in time bin *D* − 1 and later.

We learned the TRBM on 80% of responses to random bar trajectories, with *M* = 10 hidden units and *D* = 5. We inferred the model using persistent contrastive divergence with 400 epochs, minibatches of 2 responses of length 820 ms (*K* = 41 bins), momentum coefficient 0.9, and weight decay of 10^{−5}.

For computation of model statistics, we simulated the model using block Gibbs sampling. In order to avoid dependence on times out of boundaries, we used cyclic boundary conditions. Final model statistics (Fig. 3) were computed with simulations with 300 steps on responses spanning 2 × 10^{6} time bins.

A slightly different model was proposed [51] and later simplified [52], which was also called temporal restricted Boltzmann machine. Contrary to the model presented here, their model also allows for neuron-to-neuron and hidden unit-to-hidden unit interactions across different times. Furthermore, the dependence on the past takes has a slightly different form. At each time, the joint probability of neurons and hidden units is an RBM, with field vectors ** a** and

**depending on past values of neurons and hidden units. This difference in form has consequences for model computations. For example, it is not possible to compute exactly the probability of hidden states given neural responses (eq. 10), which is used for learning.**

*b*### 3 Response statistics

We measured how well models of population response could predict multiple statistics that are classically encountered in the literature.

*Firing rate*, or spiking frequency [15, 27]. It is equal to 〈*σ*_{ik}〉_{k}/Δ_{t} for neuron *i*, where Δ_{t} is the length of time bins. Here 〈.〉_{k} stands for the mean across time bins *k*.

*Pairwise correlation* [15, 30]: Pearson correlation between neurons *i* and *i*′: , where the covariance is cov(*X*, *Y*) = 〈*X*, *Y*〉 – 〈*X*〉〈*Y*〉 and the variance is var(*X*) = cov(*X*,*X*).

*Cross-correlation*, the correlation between neuron *i* and *i*′ with a delay *d* between them, *ρii′d* = corr_{k}(*σ*_{ik}, *σ*_{i′,k+d}), where corr_{k} is the correlation across time bins *k*.

In order to quantify how well a model predicted cross-correlations, we computed the explained variance of cross-correlations. We first computed the variance of cross-correlations for different time delays *d*, var_{i,i′}(*ρ*_{data,ii′d}, where var_{i,i′} is the variance across all pairs of neurons. *ρ*_{date} was computed on a testing set not used for training. We then computed the variance explained by the model:
where *ρ*_{model} is the cross-correlation predicted by the model.

On Fig. 2E, we show the fraction of explained variance, equal to the ratio between the explained variance and the variance, for cross-correlations at a single time delay.

On Fig. 3F,G, we compute the fraction of explained variance across 8 time bins. Here we grouped all crosscorrelations with multiple time delays for computation of the variance and explained variance.

*Population count*: number of spikes in the population during *L* time bins. At time *k* it is .

### 4 Likelihood

We computed the mean log-likelihood of responses for the RBM and TRBM models: where the empirical mean is computed across responses in the training or testing sets. For the RBM, we computed the likelihood for responses of length 20 ms (single time bin). For the TRBM, we computed the likelihood for responses of length 100 ms (5 time bins).

The probability of response ** σ** for the RBM, marginalized across all possible hidden states

**, has the form**

*h**P*(

**) × (1/**

*σ**Z*)

*e*

^{−E(σ)}, where

*Z*is a normalization constant called partition function, and: where

*W*_{j}is the

*j*

^{th}row of

**[48].**

*W*A similar equation exists for the TRBM. Due to temporal correlations, the probability of responses across time bins *k* = 1,…*K* also depends on responses in time bins *k* < 1 and *k* > *K*. In order to compute exactly the distribution of responses in time bins *k* = 1,…*K*, one needs to marginalize over all possible responses in other time bins. As this is intractable, we approximated the probability of response a in time bins As this is intractable, we approximated the probability of response ** σ** in time bins

*k*= 1,…

*K*using: where

*W*_{dj}is the

*j*

^{th}row of matrix

*W*_{d}. In eq. (14), we replaced the response

*σ*_{k}outside time bins

*k*× 1,…

*K*by the mean response 〈

**〉 predicted by the model.**

*σ*We computed the normalizing constant *Z* for the RBM and TRBM using Annealed Importance Sampling [53, 54] with 5000 intermediate temperatures and 5000 responses generated at each temperature.

### D Neural metrics

The response of a population of neurons consists in a series of action potentials, or spike train. We note *R* = (*t*_{in})_{in} the population response, with *t*_{in} the time of the *n*^{th} spike from neuron *i*. Neural metrics are functions that associate a non negative value to each pair of responses *R*^{(1)} and *R*^{(2)} (exponents with parenthesis are indices). As such, they are a measure of the dissimilarity between responses. In the following we present multiple neural metrics that can be found in the literature, and then introduce new metrics based on the RBM and TRBM. When a metric from the literature was only defined for single neurons, we adapt it to a population by summing the metric for each neuron.

The first three metrics are functional metrics [55]: responses are first mapped onto time dependent vectors, and the metric is defined in this functional space. The remaining metrics are defined directly on spike trains.

#### 1 van Rossum metric

The van Rossum metric is a kernel-based metric. To map a response *R* to a time dependent vector *v*, each neuron’s spike train is convolved with a kernel *H*: . We then take the Euclidean distance between convolved spike trains.

Classically *H* is a decaying exponential: *H*(*t*) = *e*^{−t/c} if *t* ≥ 0, 0 otherwise [8] with *c* a time constant. We optimized c to maximize mean response discriminability across all responses to perturbations (data not shown, time constants for all metrics were optimized in the same way), and found *c* = 630 ms here. This constant might seem large compared to time constants of metrics presented below, but they are actually not directly comparable, as they are not on the same scale. This is due to the asymmetry of *H*: the van Rossum metric still takes spike times into account in the limit of *c* infinitely large. Indeed, for large *c*, *v*(*t*) is proportional to the number of spikes that happened before *t*. On the opposite, the metrics presented below which depend on a time constant only compare the total number of spike for each neuron when their time constant is large, with no information about their timing.

A Gaussian kernel is sometimes also considered: *H*(*t*) = *e*^{−t2/2c2} [56]. Even after optimizing the timescale, it always discriminated less well than the exponential kernel. It is therefore not shown here.

#### 2 Angular metric

The angular metric uses the same vector mapping as the van Rossum metric, but measures the angle between corresponding vectors [10]:
where 〈., .〉 and |.|_{2} are the scalar product and Euclidean norm respectively: 〈*x*,*y*〉 = ∫ *x*(*t*) *y*(*t*) d*t*, .

In order to account for responses with no spike, we add an offset *α* to convolved spike trains: . We used a Gaussian kernel, optimized **α** and the time constant *c* (data not shown) and found *α* = 10^{−5} (for kernel with integral norm 1 s) and *c* = 80 ms.

#### 3 Inter-Spike Interval metric

The Inter-Spike Interval (ISI) metric measures the dissimilarity between responses inter-spike interval profiles ** ν** [57, 58]. For each neuron

*i*and time

*t*, we define

*ν*

_{i}(

*t*) =

*t*

_{i,n+1}–

*t*

_{in}, where

*t*

_{in}(resp.

*t*

_{i,n+1}) is the first spike before (resp. after)

*t*for neuron

*i*. The ISI metric is then:

We used an edge-correction in order to estimate ** ν** before the first spike and after the last [58]. The ISI metric has no parameter.

#### 4 Victor-Purpura metric

The Victor-Purpura metric [6] is an edit-length metric: the distance between two spike trains is the minimal cost necessary to transform a spike train into the other. Deleting or adding a spike costs +1, whereas moving a spike by Δ*t* has a linear cost *q*Δ*t*. We optimized *q* to maximize mean response discriminability (data not shown) and found *q* = 13 s^{−1}.

#### 5 Nearest-Neighbor metric

The Nearest-Neighbor metric measures the similarity in spike times [11]. Given two population responses and , we compute the distance between them by computing, for each spike *n* from neuron *i*, the time difference with the nearest spike in the other response: , and symmetrically for Δ^{(2)}. The distance between the two population responses is then:
where 〈.〉_{n} is the mean across spikes. We optimized *c* (data not shown) and found *c* = 50 ms.

#### 6 Event and Spike Synchronization metrics

The synchronization metrics are based on an instantaneous coincidence detector *F* [9, 58, 59]. For each spike of *R*^{(1)}, is equal to 1 if there is a coinciding spike in *R*^{(2)}, and 0 otherwise:

We compute *F*^{(2)} symmetrically. The synchronization metrics are then:
where the average is across all spikes in *F*^{(1)} and *F*^{(2)}.

For the Event synchronization metric, the time scale is fixed: *τ*_{in} = *c*. We optimized *c* (data not shown) and found *c* = 50 ms.

For the Spike Synchronization metric, the time scale is automatically adapted to the local firing rate of the responses, so it has no parameter. For a spike in with closest spike in the other response , we take:

#### 7 SPIKE metric

The SPIKE metric is based on the SPIKE dissimilarity profile *S*(*t*), measuring differences in timing of spike events [58, 60, 61]. For neuron *i* and time *t* between spike times in response *R*^{(1)}, we set a weighted average between times to closest spikes in the other response, and , defined in the Nearest-Neighbor metric:

We call *ζ*^{(2)} the corresponding average for response *R*^{(2)}. *S* is then a weighted sum between *ζ*^{(1)} and *ζ*^{(2)}:
with *ν* the previously defined inter-spike interval profile. The SPIKE metric is:

The SPIKE metric has no parameter.

#### 8 RBM metric

We first define metrics for responses in a single time bin, and then generalize to longer responses. We designed RBM metrics, such that the distance between binned responses *σ*^{(1)} and *σ*^{(2)} depends on the difference between the probabilities of hidden units conditioned by neural responses, *P*(** h**|

*σ*^{(1)}) and

*P*(

**|**

*h*

*σ*^{(2)}).

There are multiple ways to compute a difference between distributions, such as the Kullback-Leibler divergence, but we aim at finding a metric that is convenient for computation. We notice that hidden units are binary and independent when conditioned by a neural response, so *P*(** h**|

*σ*^{(u)}) for

*u*= 1, 2 is fully characterized by its mean

*P*(

**|**

*h*

*σ*^{(u)}). Therefore we chose to measure the difference between the probabilities

*P*(

**|**

*h*

*σ*^{(u)}) as a difference between mean hidden states:

**Δ**= 〈

*h***|**

*h*

*σ*^{(1)}〉 – 〈

**|**

*h*

*σ*^{(2)}〉.

The difference between those vectors was measured in two different ways, an Euclidean and a semantic metric, simply refered to as RBM metric in the main text. The Euclidean metric for the RBM is

For the semantic RBM metric, we reasoned that two states of hidden units are similar if they trigger similar neural responses. We first define a metric between states of hidden units (termed hidden states) that takes this consideration into account, and then apply it to measure the difference between mean hidden states **Δ h**.

We define a metric in the space of hidden states, such that the distance between hidden states *h*^{(1)} and *h*^{(2)} depends on the difference between the probability of neural responses they trigger, *P*(** h**|

*σ*^{(1)}) and

*P*(

**|**

*h*

*σ*^{(2)}). It can be shown from eq. (6) that if the difference between

*h*^{(1)T}

**and**

*Wσ*

*h*^{(2)T}

**is always constant across different values of**

*Wσ***, then**

*σ**P*(

**|**

*h*

*σ*^{(1)}) and

*P*(

**|**

*h*

*σ*^{(2)}) are equal. Thus if (

*h*^{(1)}–

*h*^{(2)})

^{T}

**has only small fluctuations when**

*Wσ***is generated by the RBM model, then**

*σ*

*h*^{(1)}and

*h*^{(2)}have a similar inuence on neural responses. They can have very different probabilities to happen, but when they do they co-occur with similar neural responses. We thus chose to measure the distance between

*h*^{(1)}and

*h*^{(2)}as var

_{σ}[

*h*^{(1)}–

*h*^{(2)})

^{T}

**, where var**

*Wσ*_{σ}is the variance across neural responses predicted by the model. If this value is 0, then

*P*(

**|**

*h*

*σ*^{(1)}) and

*P*(

**|**

*h*

*σ*^{(2)}) are the same. The semantic RBM metric between 2 responses is then: where

**is the covariance matrix of neural responses predicted by the model.**

*C*As we show in Fig. S2, the semantic metric is better at discriminating responses than the Euclidean one, as it is less affected by the redundancy between the parameters of the RBM. In this article we always take the semantic metric by default, unless explicitly stated.

Finally, the RBM metric between responses lasting multiple time bins is:

#### 9 TRBM metric

The RBM is a special case of the TRBM where neurons are only connected to hidden units in the same time bin. We thus define TRBM metrics so that they are consistent with the RBM metrics. We define the vector **Δ h** = 〈

**|**

*h*

*σ*^{(1)}〉 – 〈

**|**

*h*

*σ*^{(2)}〉, with indices (

*j*,

*k*) over hidden units

*j*and time bins

*k*. The TRBM metrics are then:

The later can also be written in matrix form:
with
where *C*_{d} is the cross-covariance between neural responses with delay *d*.

In the special case of a TRBM with no interaction between different time bins (*D* = 1), TRBM and RBM metrics are equivalent.

#### 10 Continuous TRBM metric

In order to define a metric based on the TRBM that does not require to binarize responses, we introduce a continuous time approximation of the semantic TRBM metric.

**Δ h** is the difference between 〈

**|**

*h*

*σ*^{(1)}〉 and 〈

**|**

*h*

*σ*^{(2)}〉, and eq. (32) measures a norm of this difference. In order to express the TRBM-based metric in a form that is convenient for expression in continuous time, we approximate this difference using a linear expansion of the sigmoid function in eq. (10): where

**Δ**=

*σ***. The semantic TRBM metric becomes: where and**

*σ*^{(1)}−*σ*^{(2)}We dropped the 1/4 factor in **Δ h**, as multiplying a metric by a constant has no effect on its discriminating properties. This can be approximated in continuous time by the Euclidean metric corresponding to the following scalar product [55, 62]:
where is a continuous time approximation for for any integer

*l*. We used a piecewise linear interpolation for remaining times. The continuous TRBM metric is then: .

### E Linear discriminability

The linear discriminability is a measure that is independent of any metric, obtained by projecting responses onto a single direction. We measured binned responses *σ*_{ref} to multiple repetitions of the reference stimulus, and responses *σ*_{Smax} to multiple repetitions of the largest amplitude of the same perturbation shape (typically 110 pm). We computed the mean response to the reference, 〈*σ*_{ref}〉, and to the largest amplitude perturbation, 〈*σ*_{Smax}〉, and projected all responses onto their difference: we call *x*_{ref} = (〈*σ*_{Smax}〉 − 〈*σ*_{ref}〉) · *σ*_{ref} the projection of a response to the reference, and *x*_{S} = (〈*σ*_{Smax}〉 − 〈*σ*_{ref}〉) · *σ*_{S} the projection of a response to *S* (when doing the projection, we recalculated the mean response by excluding the response that was being projected, to avoid over-fitting). The linear discriminability of *σ*_{S} is defined as the probability that *x*_{ref} < *x*_{S}.

Note that although this definition of discriminability is convenient because it doesn’t make any assumption about a metric. It is supervised as it requires us to know the mean response to a perturbation. Conversely, dis-criminability based on metrics can be computed for a single response to a perturbation.

During the experiment, to identify the range of perturbations that were neither too easy nor too hard to discriminate, we adapted perturbation amplitudes online using the Accelerated Stochastic Approximation algorithm [63] so that the linear discriminability converged to target value 85%.

In order to compare metrics, we formed 3 groups of responses based on their linear discriminability: low (lower than 0.95), medium (higher of equal to 0.95 and lower than 1) and high (1).

### F Code availability

The code for learning the models, computing their statistics and implementing the RBM and TRBM metrics is freely available at https://github.com/ChrisGll/RBM_TRBM.

## Ackownledgements

We thank David Schwab for insightful discussions on RBMs. This work was supported by ANR TRAJECTORY, ANR OPTIMA, ANR IRREVERSIBLE (ANR-17-ERC2-0025-01), the French State program Investissements d’Avenir managed by the Agence Nationale de la Recherche [LIFESENSES: ANR-10-LABX-65], European Commission grant from the Human Brain Project n. FP7-604102, and National Institutes of Health grant n. U01NS090501.

## References

- [1].↵
- [2].↵
- [3].↵
- [4].↵
- [5].↵
- [6].↵
- [7].
- [8].↵
- [9].↵
- [10].↵
- [11].↵
- [12].↵
- [13].↵
- [14].↵
- [15].↵
- [16].↵
- [17].↵
- [18].↵
- [19].↵
- [20].↵
- [21].↵
- [22].↵
- [23].
- [24].↵
- [25].↵
- [26].↵
- [27].↵
- [28].↵
- [29].↵
- [30].↵
- [31].↵
- [32].↵
- [33].↵
- [34].↵
- [35].↵
- [36].↵
- [37].↵
- [38].
- [39].
- [40].↵
- [41].↵
- [42].↵
- [43].↵
- [44].↵
- [45].↵
- [46].↵
- [47].↵
- [48].↵
- [49].↵
- [50].↵
- [51].↵
- [52].↵
- [53].↵
- [54].↵
- [55].↵
- [56].↵
- [57].↵
- [58].↵
- [59].↵
- [60].↵
- [61].↵
- [62].↵
- [63].↵