Abstract
A key question in theoretical neuroscience is the relation between connectivity structure and collective dynamics of a network of neurons. Here we study the connectivity-dynamics relation as reflected in the distribution of eigenvalues of the covariance matrix of the dynamic fluctuations of the neuronal activities, which is closely related to the network’s Principal Component Analysis (PCA) and the associated effective dimensionality. We consider the spontaneous fluctuations around a steady state in randomly connected recurrent network of spiking neurons. We derive an exact analytical expression for the covariance eigenvalue distribution in the large network limit. We show analytically that the distribution has a finitely supported smooth bulk spectrum, and exhibits an approximate power law tail for coupling matrices near the critical edge. Effects of adding connectivity motifs and extensions to EI networks are also discussed. Our results suggest that the covariance spectrum is a robust feature of population dynamics in recurrent neural circuits and provide a theoretical predictions for this spectrum in simple connectivity models that can be compared with experimental data.
1 Introduction
Collective dynamics in networked systems are of great interest with numerous applications in many fields including neuroscience, spin glasses, and ecological networks. Many studies have focused on how certain statistics of dynamics depend on the connectivity structure of the network, such as the population average [22] and variance [25, 10] of pairwise correlations. Although powerful and directly comparable with experimental data, these are local features of dynamics meaning that it can be estimated just from the local measurements of the activity of involved neurons. In contrast, certain global features of dynamics rely on simultaneously recorded activity of a population of neurons and cannot be inferred from just local descriptions of dynamics.
One important example of such global aspects of population dynamics is the eigenvalues of the covariance matrix which are complicated nonlinear functions of all matrix elements. These eigenvalues arise naturally when performing the widely used principle component analysis (PCA) of population activity, where they correspond to the amount of variance contained in each activity PC. Another example that received substantial recent interest [28, 37, 41, 35] is the dimensionality which can be defined based on the moments of the covariance eigenvalues. Many recent experimental studies have observed a low dimensional dynamics of neurons in the brain [36, 13], and theoretical investigations have illustrated the importance of having a low or high dimensionality on brain function and computation [12], such as encoding information [3, 28].
As the experimental techniques of making joint observations of the activity large population of neurons in networks become increasingly available, it is crucial to study how the connectivity structure of the network affects these joint aspects of population dynamics to take full advantage of such data.
In this work, we study the eigenvalue distribution (i.e. spectrum) of the covariance matrix of spontaneous activity in a large recurrent network of spiking neurons with random connectivity. We study several basic and widely used models of random connectivity, including independent and identical Gaussian distributed connectivity [39] (Section 2.1), networks with connectivity motifs [40, 31, 47, 22] (Section 2.3), and Excitation-Inhibition (EI) network (Section 2.4). Random connectivity has been a fundamental model in theoretical studies of neuronal network dynamics[28, 39, 24]. They can be motivated as minimal models to capture the highly complex, disordered connections observed in many neuronal circuits such as in cortex. Some aspects of these covariance spectra might be distinct from those under ordered, deterministic connectivity (Section 3.1).
The dynamics we consider here is simple where the co-fluctuations between neurons are described by linearizing around the steady state [27, 44]. Despite the simple dynamics and minimal connectivity model, we found the resulting spectrum has a continuous bulk of non-trivial shape and shows interesting features such as a power law long tail of large eigenvalues (Section 2.2), and strong effect due to the non-normality of connectivity matrix (Section 2.3.3). These covariance spectra highlight non-trivial population-level structures of neuronal co-fluctuations shaped by recurrent interactions that was previously unexplored.
The covariance eigenvalue distribution we focused on conveys rich and robust information of the joint population dynamics. In particular, it allows a derivation of closed form expressions for the effective dimension (previously known for the simple random iid Gaussian connectivity [10]) Furthermore, we show that the continuous bulk spectrum has the advantage over low order statistics such as the dimension measure in that it is robust to outlier eigenvalues (Sections 2.3 and 2.4 and Fig. 7).
Our analytically derived eigenvalue distributions can be readily compared to real data of population activity of recurrent neural circuits, or to simulations of more sophisticated computational models. We provide ready-to-use code to facilitates such application (Section 4.8).
1.1 Neuronal networks with random recurrent connectivity
We consider a recurrent network of generalized integrate-and-fire neurons [44] and under the common dynamic regime and assumptions used in the literature [44, 27]: we assume the network is at a steady state of asynchronous firing, that is, the neurons fires irregularly without coherent timings across the population. Under the asynchronous firing regime, linear response theory can be used to approximate closely the fluctuation of neuron spiking [27, 44]. Here denotes convolution. yi (t) = ∑k δ(t − ti,k) and Δyi(t) = yi(t) − ri is the mean subtracted spike train. is the “unperturbed” spike train in absence of recurrent connections Wij (as a matrix of connection weights W = {Wij}). Ai(t) is the linear response kernel for neuron i typically having a rise-and-decay shape. Fij(t) is the temporal kernel of the synapse. For simplicity, we assume the neurons and synapses have identical dynamics, that is, Ai(t) = A(t), Fij(t) = F (t) and are independent and identically distributed for all i. Further details of the model can be found in [22] and do not affect our results in this work as explained below.
We will focus on the structure of long time scale co-fluctuations in the network, which are described by the long time window covariance , where Cij, ΔT is the covariance of the spike counts summed over a window of length ΔT : Cij, ΔT = ⟨Δsi(t) Δsj(t) ⟩, , . For spiking neurons, Cij, T typically settles to its limiting value when Δ T > 50ms [5]. As shown in [44], the covariance matrix C = (Cij) can be derived from Eq. (1) using Fourier transform and has a particularly simple form, Here I is the identity matrix, and A − 1, AT are matrix inverse and transpose (A−T = (− 1)T). The scalar summarizes the cellular and synaptic dynamics. We define J = aW for notation simplicity and refer to J as connectivity matrix., is the long time window variance of the spike trains of an isolated neuron (assumed same for all neurons). The covariance matrix C can also be easily calculated from simultaneously recorded neuron activity data (Method). We consider generalizations beyond the long time window covariance in Sec. 2.5.
Our analysis and results starts from the covariance-connectivity relation Eq. (2), which also appears in other scenarios including alternative spiking neuron networks (Hawkes model [19, 32]), fixed point activity averaged over whitened inputs, OU process [14, 23], and structural equation modeling in statistics [1] (see details in Section 4.2).
For many biological neural networks, such as cortical local circuits, the recurrent connectivity is complex and disordered. Random connectivity is a widely used minimal model to gain theoretical insights on the dynamics of neuronal networks [39, 45]. We first and mainly consider a random connectivity where Jij ∼ 𝒩 (0, g2/N) are drawn as independent and identically distributed (iid) Gaussian variables with zero mean and variance g2/N (referred subsequently as the iid Gaussian connectivity). We will later show how to generalize to other types of random connectivity, including Erdős-Rényi random connectivity, networks with excitation and inhibition, obeying Dale’s law (Section 2.4), and with connectivity motifs (Section 2.3). The theory we derived assumes the network is large and is exact as N → ∞, and we verify their applicability to finite size networks numerically.
1.2 PCA and dimensionality
Principle component analysis (PCA) is a widely used analysis of population dynamics, where the activity is decomposed along orthogonal patterns or modes. The modes can be calculated as the eigenvectors of the covariance matrix C (Eq. (33)) and the associated eigenvalues λi are positive and shows the amount of activity or variance distributed along the modes. In this work, we focus on the distribution of these covariance eigenvalues, described by the (empirical) probability density function (pdf) pC(x) which is defined as . We also refer pC(x) as the spectrum (do not confuse with the frequency spectrum in Fourier transform). We will derive the limit of pC(x) as N → ∞ and study how it depends on parameters of the connectivity model such as connection variance g.
The shape of pC(x) can provide important theoretical insights on interpreting PCA. For example, it can be used to separate outlying eigenvalues corresponding to low dimensional externally driven signals from small eigenvalues corresponding to fluctuations amplified by re-current connectivity interactions [15] (Fig. 7). pC(x) is also closely related to the (participation ratio) dimension [28, 34]: To see D is a dimensionality measure, consider when there are d non-zero λi with same value and the rest eigenvalues are zero. Importantly, D can be calculated from the first two moments pC(x). We will also derive formulas of D in the random connectivity models.
2 Results
2.1 Continuous bulk spectrum with finite support
For networks with iid Gaussian connectivity (Section 1.1), there is one parameter g describing the overall connection strength. For stability of the fixed point and the validity of the linear response theory around it, g is required to be less than 1 [39]. The parameter σ- in Eq. (2) just scales all λi and thus is hereafter set to 1 for simplicity. The main result of this paper is the derivation of the probability density function of the covariance eigenvalues in the large N limit. In (Supplementary Materials) we show that the eigenvalue pdf is where and pC(x) = 0 for x > x+ and x < x−. The distribution has a smooth, unimodal shape and is skewed towards the left (Fig. 1C). Near both support edges, the density scales as (Supplementary Materials). The derivation of the formula is based on techniques similar to those used previously in deriving the eigenvalue distribution of J [4, 38] However, we emphasize that our result on the covariance spectrum does not have a simple relation to the well-known spectrum of the connectivity J because J is a non-normal matrix (i.e. JT J ≠ JJT). This point is further elaborated in Section 2.3.3. Although the theory is derived in the large N limit, for networks of sizes of several hundreds, our results still match well with numerical simulations (Fig. 1AB). Interestingly, the derived pdf is different from the well known Marchenko–Pastur law ([29], Eq. (28) in Methods) for the pdf of the covariance matrix of sampled iid Gaussian noise, highlighting the crucial role of the nontrivial correlations generated by the recurrent dynamics.
2.2 Long tail of large eigenvalues near the critical coupling
As g approaches the critical value of 1, the upper limit of the support x+ diverges as (1 − g2) −3 (Section 4.3 in Methods). This means there is an activity mode with diverging variance and is consistent with the stability requirement of g< 1. Note that lower edge x− is always bounded away from 0, and has a limit of as g → 1. Analyzing the shape of pC(x) for large x in the critical regime g → 1 indicates a long tail of large eigenvalues, with a power law shape (Fig. 2AB) Moreover, the power law approximation does not require g be very close to 1 and is valid as long as x− ≪ x ≪ x+ is far away from the support edges (Fig. 2C, see details in Methods and Supplementary Materials).
Because the probability density is small at the power law long tail, large eigenvalues can appear to be sparsely located (Fig. 1A) and potentially mistaken for statistical outliers. This underscores the importance of knowing the exact distribution and support edges for interpreting PCA results of population activity, which we revisit later at (Fig. 7).
Note that such a long tail is a distinct feature of correlations arising from the recurrent network dynamics. For example, for the Marchenko–Pastur law that is often used for modeling empirical covariance spectra, the upper edge of the tail relative to the mean is bounded by 4 (Methods), whereas for pC(x) (Eq. (4)) the edge relative to the mean can be arbitrarily large as O((1 − g2) −2) (see below for the evaluation of the mean). This highlights the difference between covariance generated by finite samples of noise and correlations generated by the recurrent dynamics.
The long tail of the eigenvalue distribution is also reflected by a low dimension (Eq. (3)). We show that the mean and second moment of the eigenvalues λ for large N are From Eq. (7), the dimension is In particular, the relative dimension with respect to the network size vanishes as g approaches 1 (Fig. 1D). In comparison, for the Marchenko–Pastur law (Eq. (28)) is at least .
While these low order moments can be derived from previous methods (see e.g., [10] and Supplementary Materials), our method allows for the first time the derivation of the full pdf, from which higher order moments can also be derived, for example,
2.3 Impact of connectivity motifs
We next consider generalizations of random connectivity beyond the iid Gaussian model (Section 1.1). One important feature of biological neural networks is having motif structures [40, 31], which means the frequencies of certain subgraphs or motifs are significantly different from that in an edge-shuffled network (i.e. an iid random graph with matching connection probability). We can introduce second order motifs (Fig. 3A), i.e. those composed of two edges, to the Gaussian random connectivity using an additive model (Eq. (29) in Methods, [21]) which introduce the following correlations between the edge variables Here measures the strength of diverging, converging, chain, and reciprocal motifs respectively (see Methods). The previous iid model of J (no motifs) corresponds to all being 0. Under the new Gaussian random connectivity (Eq. (29) in Methods), the diverging, converging and chain motifs are generated by adding random correlated low rank components to a full rank matrix , where is a Gaussian matrix with only reciprocal correlations. We have shown that a low rank perturbation to will not affect the continuous bulk spectrum of the covariance in the large network limit, but introduce at most a finite number of outliers (Supplementary Materials, Fig. 3BC). Therefore, for the continuous bulk covariance spectrum, we can focus on the study the case of a Gaussian coupling matrix with only reciprocal motifs. Note that the motif measures defined in Eq. (10) are correlations that are normalized by var(Jij). When focusing on the component, we need to account for the change from var(Jij) to var by using So all the other types of second order motifs do not directly affect the bulk spectrum, but channel their impact via the effective . We will from here on focus on the reciprocal only case to study the shape of the bulk spectrum and write for notation simplicity.
2.3.1 Symmetric and anti-symmetric random networks
First, we look at two special, extreme cases for the reciprocal motifs (as a correlation −1 ≤k ≤ 1): J being symmetric (Jij = Jji, k= 1) or anti-symmetric matrix (Jij = Jji, k= −1). These cases are much simpler, because J is a normal matrix so pC(x) can be derived from the well known eigenvalue distribution of J ([38]). For symmetric random connectivity, Here stability requires that . For anti-symmetric random connectivity, Here the network is stable for all g. The derivations are given in the Supplementary Materials.
From the above equations, we see that pC(x) of the symmetric random network has a power law tail analogous to Eq. (6) as g → 1/2 for large x, but with a different coefficient, The pC(x) of the anti-symmetric random network does not have a long tail as the upper limit of the support is always 1.
2.3.2 General reciprocal motifs
Now, for the Gaussian random connectivity with reciprocal motifs k = ρ (Jij, Jji), −1 < k < 1, we derived an equation for pC,g,k (x) in the large N limit based on results in [38] similarly as the iid case (Supplementary Materials). Although a closed form expression is possible using the root formula for quartic equations, it seems quite involved and is thus not pursued but left implicit to the code that we use to numerically calculated pC,g,k (x). For a fixed g, as decreases, the distribution narrows in from both sides (Fig. 4B). Intuitively, these effects are mainly due to the change of stability constrained critical gc = (1 + k) −1 with (based on the spectrum of J [38]). So the same g becomes relatively weaker comparing to gc as kappa decreases. This motivates us to compare the distributions pC,g,k (x) with the same relative coupling strength gr = g/gc = g(1 + k), which is also the maximum real part of J’s eigenvalues [38]. As shown in Fig. 4C, the effects of are now smaller comparing to Fig. 4B since the main effect on relative coupling is controlled, and the changes to the shape is the opposite: the distribution narrows as increases. As strong connections, we also observe a highly skewed long tail of large eigenvalues. Interestingly, we show that (Supplementary Materials) for −1 < kre < 1 reciprocal motifs, the critical covariance spectrum has an approximate power law tail with the same exponent as the iid random case (Eq. (6)) These shape changes of pC,g,k (x) with reciprocal motifs are also reflected by the dimension measure, for which we derived a closed form expression (Supplementary Materials) Here µ1 is the mean of the distribution. Comparing with Eq. (8), this shows the non-trivial dependence of dimension on the reciprocal motif strength. Consistent with the shape changes, the dimension increases with reciprocal motifs (Fig. 4D).
2.3.3 Strong non-normal effects at kre< 0
Here we revisit the implications of the fact that the random J is a non-normal matrix. In particular, we consider the shape of pC,g,k (x) in networks with negative reciprocal motifs, k < 0. One may naively expect that as k decreases towards −1, the shape of pC,g,k (x) will become similar to that of the anti-symmetric network pC,g,k =−1(x) (Eq. (13)). Note that pC,g,k =−1 (x) has a diverging density at the upper edge of 1 (Supplementary Materials), and can also be bimodal for g in certain range (i.e. has another peak in addition to the divergence at 1, Fig. 5A).
One may thus expect pC,g,k (x) inherit some of these features and is bimodal when k is close to − 1.
However, we find that for all k> − 1 and g < gc, there is no divergence in pC,g,k (x), and the pdf is unimodal for all but a minuscule combinations of k and g whereis very close to1 (Fig. 5B-E, the main figure in panel E is a highly zoomed-in view in order to show the bimodal region which is hard to see within the whole attainable parameter region (the inset of panel E)).
This intriguing observation is due the non-normal connectivity J. To show this, we compare C,g,k (x) to the covariance eigenvalue distribution from a network with a normal connectivity matrix Jn that has a matching eigenvalue distribution to J. Similarly we consider how changes as k < 0 approaches − 1. Note that at k = − 1, J itself is normal and . Interestingly, the normal matching shows similar features as the k = − 1 (Fig. 5F): the diverging density of pC,g,k =−1 (x) at the upper edge corresponds to a non-differentiable peak and the distribution is bimodal, both phenomena occurs in a large region of (g,k) (Fig. 10 in Supplementary Materials). The absence of such features in the actual pC(x) thus highlights the strong effect on the covariance eigenvalue distribution due to connectivity being non-normal.
2.4 Erdos-Renyi connectivity and E-I networks
The Gaussian random connectivity we have considered so far has a non-zero connection weight for all pairs of neurons with probability 1. A more biologically plausible alternative is a network with random sparse connections (Erdős-Rényi (ER) random connectivity) where each pair of neurons is independently chosen to be connected with a weight w0 > 0 at probability 0 <p < 1 (and unconnected with probability 1− p). Here we compare the dynamics of these two types of networks (dense and sparse) following the framework of [24] but in terms of the covariance spectrum.
For the all excitatory ER network, we found that the covariance matrix spectrum from the iid Gaussian connectivity (Eq. (4)) also describes the bulk spectrum of an ER network once substituting to match the variance of connections var(Jij) (similar to a replacement in [24]). The result can be understood by viewing J as (J − w0p1)+ w0p1, where 1 is a matrix of all 1 entries. The first matrix has zero mean iid entries. In light of universality results in random matrix theory [43], we hypothesize it to have the same spectrum as the matching iid Gaussian matrix as N → ∞ and the expected degree pN → ∞, which is verified numerically (Fig. 6A). The second part of J is a rank-1 matrix. As mentioned in the case with motifs, adding a low rank term to the connectivity does not affect the continuous bulk spectrum (Supplementary Materials), therefore J has a same bulk covariance spectrum described by the iid Gaussian theory (see Fig. 11 in Supplementary Materials). However, for all excitatory ER network the low rank term does affect stability [24] and creates a large outlying eigenvalue (Fig. 11A in Supplementary Materials). This means either w0 or p has to be small to ensure stability. Such a limitation can be overcome by adding inhibition [24] in subsequently discussed networks.
Adding inhibition
Having inhibitory connections (negative connection weights) is an important feature in neural circuits. In the iid Gaussian connectivity, this is implicitly modeled as the entries having zero mean on average showing the cancellation of inhibition and excitation (positive connection weights) [33, 24]. In the ER network, we can add a global inhibition, where the summed output of all neurons is fed back to each neuron with a strength of − wI < 0. This is equivalent to have a connectivity J − wI 1. Since this is again a rank-1 perturbation, this ER network with inhibition has the same bulk spectrum described by Eq. (4). Moreover, the added inhibition cancels the overall excitation and allows for larger w0 and effective g while keeping the network stable [24]. In particular, for wI = w0p this is the same zero mean matrix consider above (Fig. 6A). Similarly, stability can be attained at stronger connections in other connectivity models with inhibition (according to the general conditions of [24] simplified at linear dynamics) such as (i) all-inhibitory ER network (w0 < 0, Supplementary Materials) and
(ii) mixed E-I network where the entries are independent and identically distributed as where we, wi > 0 and satisfies the balance wepe − wipi = 0 to ensure zero-mean. By similar hypothesis of universality describe above, the covariance spectrum of J is again described by Eq. (4) with an effective g matching var(Jij), which agrees well with the numerics (Fig. 6B).
Dale’s law E-I network
Biological neuronal networks largely obey the so called Dale’s law, where each neuron can send out only excitatory connections, or only inhibitory connections depending the cell type. This requires a two population network with E and I neurons, where columns of J is constrained in sign. Here we consider a simple case [33, 24]: N/2 E neurons sending connections each with strength we > 0 to other neurons at probability p, and N/2 I neurons sending − we < 0 connections with probability p. The overall strength is balanced in this network and var(Jij) is the same for all entries. Again, we found that the bulk spectrum is well described by the iid Gaussian theory (Fig. 6C) with an effective [24]. This can be generalized to unequal fractions of E and I neurons as long as the variance of connections is kept equal (Fig. 6D). For general EI networks with unconstrained connection weights and probability for E, I neurons, the covariance spectrum differs from the iid Gaussian case, analogously as their connectivity spectra differ [33], and its characterization is left for future work.
2.5 Frequency dependent covariance
We have focused on the long time window covariance matrix. This would be especially suitable to neural activity recordings that have limited temporal resolution such as calcium imaging [46]. Temporal structures of correlation beyond the slow time scale can be described by the frequency covariance matrix (or coherence matrix) where is the Fourier transform of the spike train and z* is the complex conjugate. Importantly, the long time window covariance corresponds to C(ω = 0) (Wiener-Khinchin theorem, Supplementary Materials). Non-zero frequency covariance matrix can be calculated from neural activity data either using the formula above or by Fourier transform of the time-lagged cross-correlation functions. Importantly, there is a simple C-J relation analogous to Eq. (2) for C(ω) (Supplementary Materials).
It turns out for our main model of iid Gaussian random connectivity, the spectrum of C(ω) is simply obtained replace g with an effective which takes into account frequency dependency of neuronal (A(ω), Section 1.1) and synaptic (F (ω)) dynamics. Note, however, such a replacement by the effective g may not apply to other cases, for example networks with non-zero reciprocal motifs and there can be qualitative changes with frequency (Supplementary Materials).
2.6 Fitting the theoretical spectrum to data
Our theory on the bulk covariance spectrum can be fitted to neural activity whenever the co-variance eigenvalues can be calculated. In many settings, the value of the unperturbed neuronal variance σ2 is not known. But this can be easily addressed by scaling both the observed eigenvalues and theory to have mean equal to 1. After fitting the connectivity parameter g for the normalized eigenvalues, σ2 can then be easily estimated using the original mean. For our theoretical spectra, the mean µ of covariance eigenvalues is available in closed form (Eqs. (7) and (16)), and the scaled pdf is easily found as pR(x) = µpC(µx).
Furthermore, the recorded neural activity is sometimes normalized for each neuron (e.g. by converting to z-scores), in which case the eigenvalues of a correlation matrix instead of a covariance matrix needs to be analyzed. Interestingly, we found that for our Gaussian random connectivity models, the correlation eigenvalue distribution in the large N limit is the same as the rescaled pR(x) above. This is because the diagonal entries of C becomes uniform (thus converge to µ) for large N (Supplementary Materials).
The best fitting theoretical spectrum can be found by minimizing the L2 or L ∞ error between the empirical and theoretical cumulative distributions (motivated by methods in hypothesis testing, see Methods) with respect to connectivity parameters such as g. We note that the availability of closed form or analytic solutions of the theoretical distributions makes this optimization highly efficient.
We conclude by demonstrating fit to data with a simple example while the full application to neural data will be explored elsewhere. Consider a covariance matrix that is a sum of two components: one from the iid Gaussian random recurrent network (Eq. (2)) and another low rank part representing contributions from e.g., external inputs from two other brain areas, each with magnitude and projection weights to neurons in the recurrent network described by column vector ui. Here u1 and u2 are chosen as random vectors with unit 2-norm and σ1 and σ2 are set to be sufficiently large such that the low rank part corresponds two outlying eigenvalues [7]. We then fit the theoretical spectrum (Eq. (4)) to all eigenvalues of the total covariance matrix. The inferred g is highly accurate despite the presence of (unknown to the algorithm) outliers, and can in fact often correctly separate them out (Fig. 7).
3 Discussion
In this work, we studied the eigenvalue distribution (i.e. spectrum) of the covariance matrix of a randomly connected neuronal network during spontaneous fluctuations around a steady state. We derived the distribution in the large network limit analytically in terms of the connectivity parameters such as coupling strength and motif statistics. Our results also include closed form expressions for the dimension measure generalizing known results [10]. These results are to the best of our knowledge, novel. The new spectrum exhibits characteristic long tails of large eigenvalues at strong connections that are approximated by a power law. Knowing the exact shape and support of the bulk eigenvalue distribution can facilitate separating out outlying eigenvalues corresponding to low dimensional structure (coming from other un-modeled effect such as external input).
The shape of the bulk spectrum reflects structured amplification of the neuronal noise by the random recurrent interactions and is robust to low rank components in the connectivity or from external inputs (Supplementary Materials). The interpretation bulk spectrum corresponds to smaller eigenvalues in the PCA analysis of neural activity data, their meaning and relation to circuit connectivity, Unlike the large eigenvalues [30], the interpretation of the bulk spectrum of PCA of neural activity data has received little attention. A notable exception is a recent work [8] which studied the power law of covariance spectrum of data near criticality based on renormalization group analysis. Our theory thus provides an important benchmark to compare with experimental data and advocates the bulk covariance spectrum as a powerful joint description of collective dynamics in neuronal networks.
One limitation of The main limitation of the work is the assumed dynamic regime where fluctuations of the neuronal activity are described by linear response theory [27, 44] around a fixed point. Extensions and comparisons to highly nonlinear activity such as chaotic dynamics [39] is left for future research. Future work could also consider more general network architectures such as multiple populations of EI networks [24] and incorporating distant dependent connectivity patterns based on known cortical microcircuit architectures [6, 9, 26].
3.1 Ordered vs. disorder connectivity
We have studied the covariance spectrum under random connectivity which are used to model for complex recurrent networks. Here we ask whether certain features of these spectra are distinct results of the connectivity being random. To address this question we briefly discuss here the covariance spectra from a few widely known examples of ordered connectivity for comparison.
The simplest nonrandom connectivity model is the uniform all-to-all connected network where Jij = g/N for all i, j. The covariance matrix has one eigenvalue being 1/(1 − g2) and N − 1 eigenvalues being 1. Thus, for large networks, the spectrum consists a delta function at 1 (‘bulk’) and a single outlier. A similar situation arises when we consider a ring network [6], with long-range connections, where Jij follows a smooth shape depending on the distance of the pair of neurons on the ring (Fig. 8, see Methods). In the large network limit, the covariance spectrum has delta distribution at 1 with a number of discretely located larger eigenvalues.
We next consider the ring network with short-range connections, in particular, nearest-neighbor connections. The covariance spectrum now has a continuous bulk on an interval, but the density diverges at both edges as (Fig. 8B).
To seek further if we can find an ordered connectivity with a qualitatively similar covariance bulk spectrum as the random connectivity, we consider the multi-dimensional generalizations of the nearest-neighbor ring network, i.e., a torus (Methods). As the dimension of the torus increases, the smoothness within and at the edges of the support increases, and the covariance spectrum becomes qualitatively similar to the random case [16] (Fig. 8F). This indicates that the overall smooth density and long tail shape of the covariance spectrum is a shared property in highly connected networks with high rank connectivity matrices, including random networks and high dimensional short-range spatially invariant networks.
4 Methods
4.1 Models of random connectivity
Here is a summary of results on various random connectivity models.
iid Gaussian random connectivity Jij ∼ 𝒩 (0, g2/N): closed form pdf and endpoints Eq. (4), including frequency dependent covariance spectrum Section 2.5, and power law tail approximation Eq. (6).
Gaussian random connectivity with second order motifs Eq. (10): analytic solution and endpoints (quartic root, Supplementary Materials) and power law tail approximation (Eq. (15)). For special case of symmetric and ant-symmetric connectivity, closed form pdf Eqs. (12) and (13), including frequency dependent covariance spectrum (Supplementary Materials).
Erdős-Rényi and certain EI network Section 2.4: same bulk spectrum as the iid Gaussian case.
For all cases, the mean µ and the dimension D are derived in closed form (Eqs. (7), (8) and (16).
For simplicity, we do not require Jii to be zero (i.e. no self-coupling), but allow it, for example in iid Gaussian model, to be distributed in the same way as other entries Jij. In large network limit, since individual connections are weak (e.g.), allowing this self-coupling or setting Jii = 0 does not affect the covariance spectrum (Supplementary Materials).
4.2 Applications to alternative neuronal models and signal covariance
Although the relation between C and J (Eq. (2)) is derived based on the linear response theory of integrate-and-fire neurons, it also arises in other networked systems.
Linearly interacting Poisson neurons
This is also called a multivariate Hawkes model [19]. This is a simple model for spiking neuron networks, but is versatile to capture for example the temporal spiking correlations seen in other more sophisticated nonlinear spiking neuron networks [32, 18]. A time dependent Poisson firing rate is calculated as a filtered input spike trains sj(t) (sum of delta functions), and spikes are then drawn as a Poisson process given yi(t), Here we consider a homogeneous network where the baseline firing rate y0 and response filter A(t) is the same for all neurons.
The exact long time window spike count covariance matrix of this network can be shown to be [19] which is valid if the time varying yi(t) do not often becomes negative (for example when any negative connections Ware small compare to y). Here , Y and Y are vectors of baseline and perturbed (with recurrent connections) firing rates of the neurons respectively. If we assume that the effective connection strength aW is weak so that we can approximate Y with Y 0, then (21) becomes the same as Eq. (2) (note that for Poisson process .
Another condition that ensures a uniform Y and does not restrict weak connection is a fine-scale balance condition of W sometimes assumed in EI networks [?], This is not unreasonable to assume, for example, considering the homeostatic mechanisms of neurons.
Ornstein-Uhlenbeck process
The multivariate Ornstein-Uhlenbeck process is a classic minimal model for stochastic dynamical systems [14], Here ξi(t) is iid Gaussian white noise with variance σ2. It can be shown that the long time window (or zero-frequency, see Sec.2.5) covariance of the system is given by Eq. (2).
Fixed point over whitened input
The covariance we considered so far describes the structure of fluctuations of spontaneous dynamics without or under fixed external input, often referred as noise covariance [2]. We can also consider a signal covariance in a network of firing rate neurons Here ui is the external input to neuron i. Assume the network settles to a steady state where all neurons have firing rate ri > 0, and approximate the nonlinearity as f (x) ≈ x, then the fixed point firing rates are Now consider the network activity across an ensemble of input patterns, which has a whitened statistics [11], It is easy to see that the covariance of firing rates is given by σ2(I − J) 1(I − J) −T that is the same as Eq.(2).
We note that Eq. (25) or equivalently appears in broader contexts beyond neuroscience and is studied in the field of linear structural equation modeling (SEM) [1].
4.3 Power law approximation of the eigenvalue distribution
First note the limits of the support edges. As . For the lower edge, can be found by Taylor expansion in (1 − g2) or note that (1 − g2)3x+x− = 1. Consider a x that is far away from the support edges as g → 1, given the above, this means, Note that since x+/x− ∼ (1 −g2) −3, there is plenty range of x to satisfies the above for strong connections when g is close to 1. Under these limits, Eq. (4) greatly simplifies as various terms vanish leading to (Supplementary Materials) This explains the validity of power law approximation away from support edges. If we are only interested in the leading order power law tail in the critical distribution (i.e. g → 1− and then x → ∞), there is a simpler alternative derivation that we also use for other connectivity models (see Supplementary Materials).
4.4 Comparison with the Marchenko–Pastur distribution
The Marchenko–Pasturdistribution is widely used for modeling covariance eigenvalues arising from noise [29, 15]. It is a family with one shape parameter α. We focus on the case when the covariance is positive definite which restrict 0 < α < 1 (otherwise there is a delta distribution at 0) and the pdf is The first two moments are 1 and 1 + α, from which we know the dimension is 1/(1 + α) has a lower limit 1/2. The upper edge α+ is bounded by 4.
4.5 Gaussian random connectivity with second order motifs
Following [21], we can construct a connectivity matrix with jointly Gaussian distributed entries as where ai, bj, are zero mean Gaussian variables that have the same variance across i, j and are independent except for Such a structure and correlation introduces the second order motifs to J, The relation between these covariances and the motifs can be seen in network where Jij take binary values of w0 or 0 [47, 22]. In large networks is the number of diverging motifs. Note that EJij /w0 is the connection probability and is expected number of diverging motifs in a matching Erdős-Rényi (ER) random graph. Therefore kdiv (called motif cumulant in [23]) is proportional to the deviations the diverging motif frequency from the (iid) ER graph. K* can be normalized to correlation as in Eq. (10) by dividing var(Jij) = a2 + b2 + c2. So we can achieve various second order motif cumulants by adjusting {a2, b2, c2, k, r}.
From Eq. (29) it is clear that J can be viewed as perturbed by a rank-2 matrix aeT + ebT, a = (ai), b = (bi), e = (1, 1,…, 1)T. According to Lemma 7.1 in Supplementary Materials, the continuous spectrum of the covariance based on J is the same as that for for large N. Note, however, the normalized value of is different from J and and needs to be mapped using Eq. (11).
4.6 Deterministic connectivity
4.6.1 Ring network with short and long range connections
In a ring network, neurons are equally spaced on a circle (can be physical or functional space) and neuron i is associated with a location xi = i/N, i = 0,…, N − 1. The connection between two neurons j and i only depends on the location difference xi − xj thus is translation invariant.
For long-range connections, the connectivity has a shape determined by a fixed smooth periodic function f (x) on [0, 1), In the large network limit, the eigenvalues of the covariance has an approximate delta distribution at 1 except for a finite number of discretely located larger eigenvalues (Fig. 8A). A precise statement of this result is described in Supplementary Materials. The outlying eigenvalues corresponds to the top Fourier coefficients of f (x).
For the nearest-neighbor (NN) connectivity, only Ji−1,i and Ji+1,i are non-zero and remain fixed as N → ∞.
4.6.2 Multi-dimensional ring network
For d-dimensional ring, the neurons are equally spaced on a d-dimensional lattice which is periodic along each coordinate. We focus on the NN connectivity where each neuron is connected to 2d neighboring neurons with strength and along direction k. We show that the probability density function at both support edges scales as (for comparison, the random network edges scale as ). This means for dimension d ≥ 2, there is no singularity at the support edges (Fig. 8).
To characterize the shape of the covariance spectrum (Fig. 8C-F), we further simplify by setting (see also Supplementary Materialsfor motivations based on 1D ring) and analytically derived pC(x) (Supplementary Materials). For small dimensions d ≤ 3, there are distinct “inflection points” within the support. As d increases, this non-smooth feature becomes less evident and becomes hard to tell in empirical eigenvalue distributions from a finite size network (not shown).
4.7 Fitting the theoretical spectrum to data
For neural activity data, C can be calculated from large number of time samples of binned spike count si(t) (assuming bin size is ΔT large), For calcium imaging data, the fluorescence is approximately integrating the spikes over the indicator time constant. So we can still apply Eq. (33) by plugging in the fluorescence signal in place of si(t) to calculate the covariance C (omit the constant factor ΔT which does not affect fitting to theory, Section 2.6).
We fit the theoretical spectrum to empirical eigenvalues by finding the connectivity parameter g that minimize the error between the cumulative distribution functions . This avoids issues such as binning when estimating the probability density function from empirical eigenvalues. We numerically integrate theoretical pdf (Eq. (4)) to get its pdf. As seen below, the theoretical cdf only needs to be calculated at the empirical eigenvalues.
The cdf error can be measured under the L2 norm using the Cramer-von Mises statistic Here n is the number of samples. Alternatively, the error can also be measure under L∞ norm based on the Kolmogorov-Smirnov statistic where xi are samples. Our code implements both measures.
4.8 Code
Codes for simulations, fitting, and making all figures are available at https://github.com/huyu00/netw_cov_spectrum.
Acknowledgements
This work has been partially supported by the Swartz Program in Theoretical Neuroscience at Harvard, the NIH grant from the NINDS (1U19NS104653) and the Gatsby Charitable Foundation. YH acknowledges support from HKUST. The authors also thank Eric Shea-Brown and Stefano Recanatesi for helpful discussions.