Abstract
Neuronal coding and memory formation depend on temporal activation patterns spanning high-dimensional ensembles of neurons. To characterize these high-dimensional spike sequences, it is critical to measure their dissimilarity across different epochs (e.g. stimuli, brain states) in terms of all the relative spike-timing relationships. Such a dissimilarity measure can then be used for dimensionality reduction, clustering of sequences or decoding. Here, we present a new measure of dissimilarity between multi-neuron spike sequences based on optimal transport theory, called SpikeShip. SpikeShip computes the optimal transport cost (Earth Mover’s Distance) to make all the relative spike-timing relationships (across neurons) identical between two spiking patterns. It achieves this by computing the optimal transport of spikes across time, per neuron separately, and then decomposing this transport cost in a temporal rigid translation term and a vector of neuron-specific transport flows. This yields a suitable geometry for spike sequences. SpikeShip can be effectively computed for high-dimensional neuronal ensembles and has linear cost. Furthermore, it is explicitly based on the higher-order structure in the spiking patterns. SpikeShip opens new avenues for studying neural coding and memory consolidation by finding patterns in high-dimensional neural ensembles.
Introduction
Information in the brain is encoded by very high-dimensional “ensembles” of neurons. These neurons encode information by firing nerve impulses (“spikes”) that are discrete all-or-nothing events. Populations of neurons can produce specific spike patterns depending on sensory inputs or internal variables [1–7]. With the development of new recording techniques like Neuropixels [8], it is now possible to simultaneously record from thousands of single neurons with a high temporal resolution. These techniques offer new opportunities to uncover the relationship between high-dimensional multi-neuron spiking patterns and sensory or internal variables. They also pose unique mathematical challenges and opportunities for the unsupervised discovery of the “dictionary” of neuronal “code-words”. For instance, how do we measure the similarity between two multi-neuron spiking patterns in terms of their temporal structure (spike sequence)?
In general, any notion of information encoding relies on the construction of a distance or dissimilarity measure in an N-dimensional space. For example, the distance between binary strings can be measured using the Hamming distance, which is an essential mathematical construct in error-correcting coding. In the brain, the distance between two multi-neuron spiking patterns is conventionally measured in two steps: (1) By computing the number of spikes/sec (firing rates) for each neuron in a certain time window; and (2) computing e.g. the angle or Euclidean distance between multi-neuron rate vectors. Using this method, it has been shown for example that high-dimensional neural ensembles span a low-dimensional manifold that relates to stimulus or behavioral variables in a meaningful way [9, 10]. However, the computation of firing rates disregards potentially rich information contained by the precise temporal order in which spikes are fired by multiple neurons (e.g. neuron i firing at time t and neuron j firing at t + τ). For instance, we expect that any time-varying sensory stimulus (motion) or action sequence may be encoded by a unique multi-neuron temporal pattern of spiking. It has been shown that multi-neuron temporal sequences encode information about sensory stimuli or are required for the generation of complex motor patterns like bird songs [1–7, 11]. Temporal sequences may also be critical for memory formation, because the plasticity rules in the brain are highly sensitive to the temporal order in which neurons fire spikes [12–14].
We have previously developed a dissimilarity measure between multi-neuron temporal spiking patterns called SPOTDis (Spike Pattern Optimal Transport Dissimilarity). This measure is defined as the minimum energy (optimal transport) that is needed to transform all pairwise cross-correlations of one epoch k into the pairwise cross-correlations of another epoch m. We showed that, due to employment of optimal transport, SPOTDis has several attractive features (see Results) [15], and can be combined with unsupervised dimensionality-reduction techniques like t-SNE or clustering techniques like HDBSCAN to characterize the state-space of multi-neuron temporal sequences [15]. However, the SPOTDis dissimilarity measure has two principal weaknesses: (1) Its computational complexity, for comparing two time epochs, is , where N is the number of neurons. This becomes a major problem for computing an M × M dissimilarity matrix (for M time epochs) for thousands of neurons. (2) It relies exclusively on pairwise spike-timing relationships (i.e 2nd order correlations), which means that it might not be sensitive to higher-order correlations in spiking patterns; in fact we provide some examples demonstrating that SPOTDis does not form a proper geometry for spike sequences. This relates to the fact that SPOTDis does not solve the optimal transport problem for the entire spike pattern, but only for neuron pairs separately.
Here, we develop a dissimilarity measure for multi-neuron spiking patterns called SpikeShip, which has linear computational complexity of . We achieve this by (1) computing the minimum transport cost of spikes for each spike train separately, and (2) discounting a global translation term in the transport flow across neurons. SpikeShip can be interpreted as the optimal transport cost to make all spike-timing relationships between two different multi-neuron spiking patterns identical; i.e. it solves the optimal transport problem for the entire spiking pattern. It is thus sensitive to the higher-order structure of spike trains.
Results
Our overall goal is to develop a dissimilarity measure between multi-neuron spiking patterns that is exclusively based on the temporal order of spiking (i.e. not on the spike count). Suppose that we have measured the spikes from N neurons (“spike trains”). Spike trains are divided into M epochs of length T (measured in seconds or samples). Epochs could be defined by e.g. trials (e.g. stimulus presentations) or sliding windows [15]. The problem is to find a dissimilarity measure with the following properties [15]:
The measure should depend on the temporal order of firing across neurons, but not on the spike count, although the dissimilarity measure below can be generalized to incorporate spike count differences.
If two spike patterns are identical in terms of cross-neuron spike timing relationships (i.e. temporally translated version of one another), then the dissimilarity measure should equal zero.
The measure does not require binning or smoothing, and is based on the exact timing of the spikes.
It should allow for the discovery of more patterns than the number of neurons, when used in combination with clustering techniques like HDBSCAN.
It should measure dissimilarity in a continuous way and avoid the problem of “combinatorial explosion”. This problem occurs with methods that search for exact matches in spiking patterns, if the number of neurons is large.
Previous work: SPOTDis
[15] developed a dissimilarity measure called SPOTDis, which is based on optimal transport theory. We briefly summarize the computing steps of SPOTDis (see Methods):
Compute cross-correlation histograms for all N2 pairs (i, j) of two neurons, . These cross-correlations capture the information about the spike-timing relationships across all pairs of neurons.
Normalize these histograms to unit mass, which is needed for the optimal transport measure, and removes the effect of spike count.
Compute, for each pair of neurons (i, j) and pair of epochs (m, k), the minimum transport cost (Earth Mover’s Distance) to transform into . This yields a dissimilarity measure denoted Dij,km.
Compute the average of this dissimilarity measure across all pairs of neurons, yielding a dissimilarity measure Dkm called SPOTDis.
Using simulations, we showed that SPOTDis can be used to find distinct clusters of multi-neuron spiking patterns in an unsupervised way [15].
As discussed in [15], SPOTDis satisfies the six requirements listed above. In particular, note that the EMD can be computed based on exact spike times and does not require binning or smoothing [15]. The use of the EMD for multi-neuron spike train dissimilarity has several additional advantages, as explained in [15]: The EMD is a symmetric and metric measure of similarity between two probability distributions (unlike the KL-divergence). In contrast to KL divergence, it does not quantify whether two distributions are overlapping in the same bins; rather, it examines how far distributions are shifted away from each across bins in a metric space (in this case: time). Different from KL, EMD is well behaved both when comparing continuous and discrete (sum of delta functions) distributions. Further, the EMD does not rely on the computation of a measure like the center of mass or the peak time of firing activation, but can deal with multi-modal spiking patterns [15]. In other words it does not require any prior specification of features. Finally, as shown in [15], SPOTDis is highly noise robust (i.e. to insertion of noise spikes or spike-time jitter) because of the use of EMD.
SpikeShip
There are two main problems with the construction of SPOTDis: First, its computational complexity scales as , where n is the average number of spikes per epoch. Note that this computation typically needs to be carried out M 2 times in order to create a dissimilarity matrix for all epoch-to-epoch comparisons. This means that in practice, computation of SPOTDis might not be feasible when N is on the order of thousands, which is nowadays achievable with modern recording technologies. Second, as discussed in [15], SPOTDis is based exclusively on the pairwise cross-correlations between neurons. Hence, it may be insensitive to the higher-order correlations in the spike trains.
To address these problems we introduce the SpikeShip measure (see Methods). SpikeShip yields a dissimilarity between multi-neuron spike train patterns that has similar characteristics as SPOTDis. But, it is has the following desirable properties:
Its computational complexity grows linearly with the number of neurons and spikes, i.e. .
By construction, it is sensitive to higher-order spike-timing relationships.
It allows for the decomposition of transport cost into a temporal rigid translation term and neuron-specific flow terms.
SpikeShip is computed in two steps (see Methods, Figure 1):
Compute the optimal transport flow to transform a spiking pattern in epoch k into the spiking pattern in epoch m. This flow vectors are computed for each neuron separately (i.e. in the spike raster plot, mass from spikes can be moved horizontally, but not vertically). This is performed in two steps: (i) Assign a mass to each spike, such that the total mass for the ith neuron in epoch k and m is equal; (ii) compute for each spike, how its mass should be shifted in order to make the spike trains in the window T completely identical.
By using an expression for a global temporal translation term (across neurons) and subtracting this from all the spike shifts, we minimize the overall transport cost. This transport cost amounts to the amount of energy to shift the individual spikes such that all spike-timing relationships across neurons become identical.
When we compute the EMD in step 1, we can compute the total transport in discrete steps, shifting mass from a spike in pattern k to mass of a spike in pattern m. This can be algorithmically solved, for each neuron, in approaximtely order . We donate the discrete steps ci,u with moved mass wi,u, such that ∑u wi,u = 1. From these flows computed in Step 1, we can then subtract a global rigid translation term. This uniquely yields the minimum transport cost to transform the multi-neuron spiking pattern in epoch k such that its spike-timing relationships are identical to another pattern m. In the Methods section we state our main result, namely that the optimal transport flows are given by
Here gmin is the weighted median across all the original flows ci,u with associated mass wi,u. Thus we can decompose the transport flow in two terms: (1) an optimal global shift between two epochs, shared across all neurons; and (2) an optimal neuron-specific transport flow based on the EMD method. We then define SpikeShip (see Methods) as
Here is the set of all neurons that fired a spike both in epoch k and m. The weight , effectively assigns an equal weight to each neuron that contains at least one spike. The algorithm to compute SpikeShip has computational complexity , because the weighted median has complexity (Figure 2). This result is, in a way, surprising: One would expect that for a dissimilarity measure that is based on the relative spike-timing between neurons and is sensitive to higher-order correlations, we would need a computational complexity of at least .
Relationship between SpikeShip and SPOTDis
We show in the Methods section that there exists an analytical relationship between SpikeShip and SPOTDis for a specific class of sequences: Suppose that (1) in each temporal sequence, a neuron is active in only in a small part of the time window T (or fires only one spike), and (2) that the temporal sequence is uniformly spanned across neurons (i.e. a uniform distribution across neurons). The relationship between SPOTDis and SpikeShip (see Figure 2) then equals
Thus, for these kind of sequences, the total transport cost Fkm, which equalizes all the higher-order correlations in the spike patterns, can be related to the transport cost over all the pairwise spike-timing relationships (SPOTDis). However, an analytical relationship may not exist for the case of multimodal spiking patterns or other types of distributions (see e.g. Example 1).
SpikeShip is a more sensible dissimilarity measure than SPOTDis because it is explicitly sensitive to higher-order structure in the spike train. In addition, it is faster. The difference between SpikeShip and SPOTDis can be further illustrated with the following simple examples.
Example 1
Suppose that we have a vector of spike times and . We now have cost and . The SPOTDis in this case equals
Consider now another vector of spike times and . We now have cost and, different from above, .
The SPOTDis in this case equals
In this case, the mover costs on the 2nd order pairwise timing relationships (SPOTDis) are the same for both sequences; however, SpikeShip correctly reflects the increased mover cost that is needed to match the two patterns to the zero pattern. We can consider further the distance between the two sequences. In this case, SpikeShip equals . However, SPOTDis now equals . Thus, SPOTDis assigns a disproportionally large transport cost between the first and the second sequence as compared to the (0, 0, 0, 0) sequence 3.
Dissimilarity matrix and cluster visualization
We compute SpikeShip to detect unique, recurring spiking patterns generated as in [15] (Figure 4). The dataset contains P = 7 patterns across N = 100 neurons and M = 360 epochs. Low-dimensional embedding was performed with t-SNE as in [15] (Fig 4B). Clusters were derived using HDBSCAN and are plotted in a 2-dimensional space. These matrices were computed using a workstation with 12 CPU cores (DUAL Xeon E5646 @ 2.40GHz). The computation time for SpikeShip and SPOTDis dissimilarity matrices were 28.2 sec and 20186.2 secs (5 hours, 36 minutes, and 26.2 secs), respectively. This is a speedup of about 700 times (which is higher than N = 100 due to presence of multiple spikes).
Discussion
We studied the problem of measuring the dissimilarity between two multi-neuron spiking patterns based on the relative spike-timing relationships across neurons (i.e. firing order). We developed a new measure called SpikeShip. SpikeShip solves the problem of optimally transporting the spikes of individual neurons, such that the global pattern of spike-timing relationships becomes identical between two epochs. Intuitively, one would think that such a measure has computational complexity of at least , but we show that it can be computed with computational complexity; this is a major improvement over our previous work [15]. We show that a dissimilarity between two spiking patterns can be decomposed into neuron-specific flows and a temporal rigid translation term. Compared to our previous measure SPOTDis [15], SpikeShip is not restricted to the 2nd order correlations, but is based higher-order structure in the spike train.
Our technique can also be used to align temporal sequences in an unsupervised way if there is a global jitter between spike patterns. This achieves a similar goal as a recent study in Neuron [16], described as “warping”; however, the method in our paper does not require any parameter estimation; furthermore, it can also be used to align trials corresponding to different stimuli or behavioral conditions. For example, suppose that we want to analyze the sequence of neural activation in relationship to some event, e.g. freely moving your arm. Suppose that we do not know the exact onset of the neuronal sequence in relationship to moving your arm, and we record multiple realizations of this event. With SpikeShip, we can immediately decompose the transport cost between any two trials in terms of a global translation term (which is a non-linear computation) and a neuron-specific shift. This allows one to align trials that were recorded in different conditions, e.g. moving your arm left or right. And it yields, in essence, a state-space characterization of temporal trajectories (e.g. related to different movements), rather than a trajectory through a state-space of firing rate vectors.
SpikeShip opens new avenues to study temporal sequences in high-dimensional neural ensembles. Recent technological developments now allow for recordings of thousands of neurons simultaneously, either using electrical recordings or two-photon imaging [8]. The technique developed here is applicable to both kinds of data, due to linear computation time. Application of SpikeShip to such kind of data might generate important insights into the role of temporal sequences in sensory coding and memory consolidation.
Materials and methods
Previous work: SPOTDis
We previously defined a dissimilarity measure between multi-neuron spike train patterns called SPOTDis [15]. We give a brief overview of the definition of the SPOTDis measure; for a detailed account we refer to [15]. Suppose we record spike trains of N neurons in M separate epochs (e.g. trials); we refer to these as “spike patterns”. SPOTDis was defined as follows: We first compute, for all M epochs separately, a normalized cross-correlation function for all N 2 pairs of neurons, where is the cross-correlation function (or cross-covariance), and si,k(t) and sj,k(t) are the spike trains of neurons i and j in the kth trial. The normalization of the cross-correlation function is necessary to avoid an influence of firing rate fluctuations, and to compute the EMD. Next, we compute the Earth Mover Distance (EMD) between the normalized cross-correlograms and for each (k, m). This yields M(M − 1)/2 × N(N − 1)/2 EMD values Dij,km. In [15], we used the L1 norm to measure dissimilarity on the time axis, but the L2 norm could be used as well. After computing the EMDs between each pair of epochs for each neuron pair separately, we compute SPOTDis as the average EMD across neuron pairs,
Here, , i.e. the set of all pairs of neurons that are active in both epochs k and m. We ignored the other neuron pairs (i.e. not contained in this set) for computing the SPOTDis, because it avoids assigning an arbitrary value to the EMD when there is no information about the temporal relationship between the neurons (i.e. when we do not have any spikes for one neuron in one epoch).
Derivation of SpikeShip
A major problem with the computation of SPOTDis (for all pairs of epochs) is that it has computational complexity of . Furthermore, because it is based on 2nd order correlations, it is not explicitly sensitive to higher-order correlations. Here we derive a new dissimilarity measure, called SpikeShip, which has computational complexity for one pair of epochs. To derive this measure, we first consider the simplified case where each ith neuron fires one spike for all M epochs.
SpikeShip for single spike per epoch
Let be a vector of flows for each neuron in epoch m. In other words, f1,km is the shift of the spike fired by the first neuron in the mth epoch. The total moving cost equals
The problem statement is to find a flow vector such that after moving the spikes, the resulting spike train patterns are identical between epoch k and epoch m, in terms of the full matrix of spike timing relationships. We wish to find the flow vector that satisfies this constraint with minimum cost Fkm.
Example
Consider the following example: Suppose there are two epochs for N = 6 neurons with spike times and . We will show that the flow vector with minimum cost, such that spike patterns have identical temporal structure, equals (Figure 1).
More formally, let tm,i be the timing of the spike for the ith neuron in the mth epoch. We denote the spike times after (post) moving them in epoch m as ∀i, where we omitted the subscripts k, m from the variable fi,km for simplicity. The constraint that all the across-neuron spike timing relationships should be identical after moving implies that
In other words, the delay between two spikes from two different neurons (i, j) should be identical between pattern k and m after moving the spikes. Substituting based on Eq. 9, this can be expressed as
Let ci be the shift in spike timing for each neuron in epoch m, such that the spike train patterns become identical,
Note that with this definition of ci, the equation holds for all (i, j). For all i, we can express fi as a function of the shift ci, such that
We wish to solve under the constraint of Eq 11.
From Eqs 11, 13 and 14 it follows that
Hence the equation gi = gj holds for all (i, j). Thus, we can rewrite Eq. 27 to to find a global shift that minimizes the L1-norm of the residuals. The solution to this this equation is
Thus, our main result is that the original shifts between the two spiking patterns can be written as the decomposition for all i, i.e. the optimal transport in terms of neuron-specific shifts and a global temporal rigid translation term. Thus, the optimal transport between two spiking patterns (e.g. after stimulus onset) can be decomposed into the optimal transport in terms of neuron-specific shifts and a global temporal rigid translation term.
Global shift definition for multiple spikes
We now consider the case where in each epoch, every neuron fires a variable number of spikes. We will show that a similar derivation for SpikeShip can be made based on the weighted median. Let ni,k and ni,m be the number of spikes for the i-th neuron in epoch k and m. In the SPOTDis, we normalized the cross-correlations to unit mass, such that each spike effectively gets the same weight. For SpikeShip, we will also assign a weight to each spike, such that the total weight per neuron in the computation of SpikeShip is equal. In order to do so, we first find the smallest common multiple of the spike counts for each neuron, denoted . For instance, if ni,k = 2 and ni,m = 6 then the least common multiple equals 6. We now replicate each spike and times. Each spike will now have a weight of . (Note that in the actual computation, we do not replicate the spikes in practice, but use an algorithm similar to the one detailed in [15]). Then, based on the optimal transport cost (EMD), we obtain shifts ci,u for the u-th spike of the i-th neuron, . Note that computation of the EMD amounts to shifting the most left-ward spike (i.e. first spike) out of spikes in epoch m to the most left-ward spike in epoch k [15].
We use a similar derivation as the one above. Let tm,i,u be the timing of the u − th replicated spike for neuron i in the mth epoch, for all . We denote the spike times after (post) moving them in epoch m as
∀(i, u), where we omitted the subscripts k, m from the variable fi,km,u for simplicity. The constraint that all the across-neuron spike timing relationships should be identical after moving implies that
In other words, the delay between two spikes from two different neurons (i, j) should be identical between pattern k and m after moving the spikes. Substituting based on Eq. 20, this can be expressed as
There is one additional constraint, namely
In other words, all the pairwise relationships after moving within the same neuron should be identical.
Let ci,u be the shift in spike timing for each neuron in epoch m, such that the spiking patterns in the window become identical,
Note that with this definition of ci,u, the equation holds for all (i, j, u). For all i, u, we can express fi,u as a function of the shift ci,u, such that
Define .
We wish to solve under the constraints of Eq 22, 23 and 25.
Given these two constraints, the equation gi,u = gj,u holds for all (i, j, u). Thus, we can rewrite Eq. 27 to to find a global translation term that minimizes the L1-norm of the residuals. The solution to this this equation is the weighted median g,
Now average mover cost, SpikeShip, equals where is the set of neurons that are active in both epochs k and m. Thus, we derive the result that the original shifts between the two spiking patterns can be written as the decomposition for all (i, u), i.e. the optimal transport in terms of neuron-specific shifts and a global translation term. Thus, the optimal transport between two spiking patterns (e.g. after stimulus onset) can be decomposed into the optimal transport in terms of neuron-specific shifts and a global temporal rigid translation term. This computation has linear complexity . We base the computation of the weighted median on [17, 18] and an adaptation of the Robustats [19] Python library. In practice, the computation of ci,u (optimal transport per neuron) can be efficiently performed not by replicating the spikes to the common multiple integer (used in derivation above), but by using a similar algorithm as in [15].
Relationship SpikeShip and SPOTDis
We now derive an analytical relationship between SpikeShip and SPOTDis, for the case of uniform sequences in which each neuron fires only at one time point of the sequence; note that this result does not generalize to more complex distributions, or the case of multiple spikes. We note that the pairwise distance among the cross-correlation for four spikes (neuron i and j, epoch k and m) equals |(tm,i − tm,j)−(tk,i − tk,j)| = |ci − cj|. For simplicity we define SPOTDis over all N2 elements (note the distance of an element with itself equals zero) and assume that all neurons fire a spike, without loss of generalization. From Eq. 31 we can rewrite the SPOTDis as
We have as expected value . Because we defined fi = ci−gmin, where , we know that the median of our distribution is zero. We can thus assess the relationship between SpikeShip and SPOTDis for different distributions of fi. We can divide the set of N (assuming N is even, without loss of generalization) neurons into two sets, namely and , with and . For the combinations (i, j) for which and we obtain the contribution to SPOTDis as where the factor reflects the probability of fi and fj having opposite signs. Thus, the term does not depend on the specific distribution of the spikes.
For the combinations (i, j) for which and the expected distance depends on the distribution of fi. If we assume that the different spike patterns each uniformly cover a window length of 2T, then we know that fi will have a triangular probability distribution, i.e.
We now obtain
Note that for this distribution we also have . Hence,
Thus, the average amount of transport on the individual spikes is smaller than the average transport on the pairs of spikes.
Authorship Contributions
Conception and problem statement: BSG, FPB, MV. Data analysis and simulations: BSG. Mathematical analysis: BSG and MV. Supervision: MV. Writing of main draft: BSG and MV.
Acknowledgements
This project was financed by the BMF (Bundesministerium fuer Bildung und Forschung), Computational Life Sciences, project BINDA (031L0167).