Model

We consider networks of N leaky integrate-and-fire (LIF) neurons. The sub-threshold membrane voltage for neuron j is given by (2) where τ_m = C_mR_m is the membrane time constant, C_m is the membrane capacitance, R_m is the membrane input resistance and I_j is the total time-varying synaptic and external input current. Once the membrane voltage passes a threshold V_th a spike is emitted and the membrane voltage is reset to V_reset.

The challenge is to connect the synapses of the network is such a way so that the network becomes a (D-dimensional) line attractor capable of retaining the value of D latent variables. That is, how should the network be connected so that the input currents I_j (synaptic or otherwise) to each neuron contributes to keeping the latent variables stable?

The general strategy we use for doing this is illustrated in Fig 1B. For each latent variable we create a corresponding latent readout-node that decodes the variable from the network activity by taking a linear combination of the synaptically filtered spike trains: (3) where is the set of weights, u_j(t) = H * ∑_m δ(t_jm − t) are the filtered spike trains, H is a synaptic kernel and t_jm is the time of the mth spike from neuron j.

We can use the read out latent variables x to construct an input current to the neurons that feeds the latent variables back into the network: (4) where is an arbitrarily chosen matrix (Fig 1B) and I_fb is vector notation for one part of the synaptic currents to the neurons (Eq 2). This artificial construct can in turn be used to find a set of synaptic weights: note that the decoding (Eq 3) and the feed-back (Eq 4) steps are both linear operations, and can thus be combined into one single matrix multiplication (5) From this, we identify as a matrix of synaptic weights and a tentative solution to the problem of how to connect the network (for a more detailed derivation of this result, see Abbott et al. [10]).

In particular, a network connected in this fashion has a basic structure needed for retaining D latent variables once they have been set. However, several more details have to be determined before implementing this idea in practice. The most crucial issue is how to find the decoding weights Φ in Eq 3. Moreover, we need to determine whether any additional synapses are required in addition to the ones indicated by W in Eq 5. In this paper, we consider three different strategies for finding Φ and designing the connections.

First, we consider the FORCE learning rule [12] which builds upon the liquid state machine and echo state networks [11, 17, 18]. Following this idea, the synapses of the network belong to two sets. The synapses of the first set are chosen randomly, but are normalized so that the input to each neuron is loosely balanced, which makes the network dynamics chaotic [19]. The second set of synapses directly corresponds to the matrix W above. Thus, for FORCE networks, the current in Eq 2 is given by (6) where is the matrix corresponding to the first set synapses and W = KΦ correspond to the second set as above. The first matrix Ω and the encoder weights K are both fixed while Φ are subject to an online learning rule (see [12] and Methods).

Second, we consider the Neural Engineering Framework (NEF) [13, 14]. Unlike the FORCE algorithm, NEF does not require any extra synapses in addition to ones implied by W. Instead, each neuron receives an additional stationary driving current so that the total current becomes (7) Similar to the FORCE rule, in the NEF W = KΦ where K is fixed and Φ is learned. In the NEF however, Φ is learned in batch. That is, (8) where the expected value is taken over all permitted values of x_target and the corresponding filtered spike trains u (see [13] and Methods).

Finally, we consider the Efficient Coding framework [15, 16]. Similar to the FORCE learning, there are two sets of synapses, a set for the reservoir and another set for the recurrent connections. In contrast to FORCE learning and the NEF, no learning is required in the Efficient Coding framework. Instead, the readout weights are fixed to (9) where λ_syn = 1/τ_syn. At the same time, the reservoir weights are fixed to (10) rather than being chosen randomly. Finally, the reservoir synapses are assumed to be instantaneous, i.e. they have a time constant of 0. The input current to neurons are given by (11) where W = λ_syn KK^T, and u_{τ = 0}(t) are unfiltered spike trains.

The neural modes and the decoders are both determined by the encoders

In Fig 2 we demonstrate the ability of the three frameworks to retain a value once it has been set. We implement a network with N = 1000 neurons and D = 2 latent variables for each of the three frameworks mentioned above. We verify the implementations using the following protocol: every 500 ms we artificially fix the two latent variables x₁ and x₂ to some arbitrary values during 100 ms (blue areas in Fig 2A, 2D and 2G) and then let the activity evolve freely for the next 400 ms. As intended from the design of the networks, they retain the variables once they have been set, albeit with varying degrees of noise and drift.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The encoders K determine the decoders Φ^T and the neural modes Z.

(A) Example of a simulation of a spiking network with the FORCE learning rule. The two traces show the two latent variables. During the periods shaded blue, the latent variables are fixed. The spike times of 50 (5% of the) neurons are shown in the background. (B) The fraction of spike rate variance explained by each principal component. (C) The generalized correlation (see Methods) between the encoder matrix K, decoder matrix Φ^T, principal components Z_PCA and loading matrix Z_FA. (D, E, F) Same result for the Neural Engineering Framework. (G, H, I) Same result for the Efficient Coding framework.

https://doi.org/10.1371/journal.pcbi.1007074.g002

For all three frameworks, the spiking pattern is largely determined by the latent variables. In particular, note that the spiking pattern (background of Fig 2A, 2D and 2G) changes considerably when the values of the latent variables are changed. These changes in firing pattern are due to the divergent connection K from the latent variables x to the input currents I (see Eq 4 and Fig 1B), which are created so that each neuron receives an input that is a linear combination of the latent variables. We expect the firing rates to approximately reflect these input currents, albeit with some distortion due to auxiliary synapses, threshold nonlinearity in the neuron and binning to estimate the firing rate.

To verify this, we estimated the firing rate in Fig 2A, 2D and 2G by counting the number of spikes from each neuron in each consecutive 50 ms interval. We then applied Principal Component Analysis (PCA) on the time series of spike bins. As can be seen in Fig 2B, 2E and 2H for all our networks two principal components explained the majority of the variance. We then calculated the cosine of the mean principal angle between the subspace spanned by the first two principal components Z_PCA and the subspace spanned by the columns of the encoder matrix K (this is a generalization of correlation, see Methods) and found a high degree of similarity (cos ϕ ≈ 0.7) for all three frameworks (Fig 2C, 2F and 2I). We also observed a similar degree of similarity with the loading matrix Z_FA from Factor Analysis (FA)—indeed, the subspaces found by FA were almost completely equal to the the subspaces found by PCA. Equating the principal components with neural modes we conclude that neural modes are determined by the encoder matrix K. Note that all the three frameworks allow us to choose K arbitrarily. Thus, they provide a way to construct networks where we can freely select not only the number of neural modes (i.e. dimensionality), but also the specific mode vectors.

The fact that the neural activity is determined by the neural modes further influences the readout weights Φ. Although the specific algorithm for determining Φ differs between the three frameworks, they all have in common that Φ should be an accurate linear transform from the spike trains to the latent variables. If the neural modes Z are orthonormal, a latent variable can be read out by taking the inner product between the respective neural mode and the neural activity: x_i ≈ z_i ⋅ u. Therefore, we expect the readout weights to be Φ^T ≈ Z (where the columns of Z span the same space as Z_PCA and Z_FA). For a more formal treatment of this argument, see Salinas and Abbott [20].

Given that Φ^T ≈ Z and Z ≈ K it is safe to assume from transitivity that (12) For the three frameworks under consideration here the correlation between Φ^T and K is in fact even larger than between Z and K or between Z and Φ^T (Fig 2C, 2F and 2I). For FORCE learning and the NEF, this is likely because while Z is found to be the directions of maximal variance, Φ^T is found by supervised learning so that the readout match the “real” latent variables as close as possible. For Efficient Coding, the correlation between K and Φ^T is exactly 1 because of Eq 9.

Synaptic modifications required for inside- and outside-manifold perturbations

Now we use the previous networks to address the main issue of this paper: why are some perturbations of the neural modes much easier to learn than others? We know that the neural modes Z ≈ K for the type of networks we are studying here. Thus, if Z is to be perturbed to perform the task, a corresponding perturbation must be applied to the encoders K.

Sadtler et al. [4] characterized perturbations of the neural modes into inside-manifold perturbations and outside-manifold perturbations. An inside-manifold perturbation is restricted so that the subspace spanned by the neural modes (the intrinsic manifold) is preserved. In other words, even though the neural modes have been changed, the part of the neural state space that is allowed is unchanged. Mathematically, the perturbed neural modes of an inside-manifold perturbation are given by (13) where . Here, we assume that the inside-manifold perturbation is defined to not stretch or skew the subspace, and therefore that is an orthogonal matrix. We refer the treatment of non-orthogonal matrices to S1 Text.

Assuming the Z ≈ K ≈ Φ^T holds for any network, it will also hold for the permuted network, i.e. , we can expect the corresponding changes to encoder and readout matrices to be (14) For all three frameworks the set of (learnable) synaptic weights are given by W = KΦ. Thus, the new set of synaptic weights after the perturbation is: (15) where follows from orthogonality. Changing the synaptic weights from W to would therefore only require minor modifications, if any.

This result does not hold for outside-manifold perturbations. Such a perturbation can be written as (16) where is also an orthogonal matrix, although with many more elements than (N² vs D²). Following the same steps as above, one arrives at (17) In general, is not equal to W and therefore learning likely requires substantial synaptic changes and hence extensive learning.

To verify this derivation, we created three new networks similar to the ones in Fig 2. Keeping all other parameters equal, we permuted the two columns of K to mimic inside-manifold perturbations, i.e. we applied Eq 14 with (18) We then learned using the respective framework, and calculated . For comparison, we created an additional three networks with the same parameters as before, but where the rows of K were permuted by switching the first 500 rows with the last 500. That is, we used (19) to mimic outside-manifold transformation and learned and as before. We thus have examples of W, and for the three frameworks. In Fig 3, we show that the element-wise correlation is high between W and (inside-manifold permutation, light green) but close to zero for W and (outside-manifold permutation, red). A high correlation indicate that only small changes to synaptic weights are required to change the neural modes within the intrinsic-manifold suggesting that this change may be easier for a biological network to learn. Note that correlations do not capture global additive or multiplicative scaling of the weights. Given that the networks have otherwise identical parameters before and after perturbation, we have no reason to expect any such scaling their weights. Nevertheless, to avoid this possibility, we also calculated the Frobenius norm which also showed a very similar pattern (S1 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Inside manifold perturbations do not require big changes of synapse weights.

(A) Synapse weights (elements of W) from one network instantiation of FORCE, compared to re-learning with the permuted encoders (elements of ). (B) Same as A, but for an outside-manifold permutation (). (C) The element-wise correlations between old (W) and new synaptic weights for different types of permutations (see main text for definitions) for all three frameworks. Error bars indicate standard deviation over 30 network instantiations. Note that orthogonal inside manifold perturbations (“Permutation” and “45 degree rotation”) require almost no changes to the weights, non-orthogonal inside manifold perturbations (“Gaussian elements” and “Same element”) require some changes and the outside-manifold permutation requires as much change as a network with unrelated encoders. See also S1D Fig.

https://doi.org/10.1371/journal.pcbi.1007074.g003

For further comparisons, we introduced four additional inside manifold perturbations: 1. no perturbation at all, just restarting the learning algorithm with the same K (Fig 3C, light blue), 2. an orthogonal inside manifold perturbation given by the following transformation (Fig 3C, green): (20) 3. a non-orthogonal perturbation matrix where all four elements were independently drawn from (Fig 3C, dark green), a non-orthogonal matrix where all four elements were equal (to a random value) (Fig 3C, darker green). In each of these different types of inside-manifold perturbations, element-wise correlation between W and was high, further suggesting that inside manifold perturbation required relatively small change in synaptic weights. By contrast, correlation between corresponding to a network where K was redrawn independently and W was almost zero (Fig 3C, dark red). These results were independent of the way we measured the similarity between W and (S1 Fig).

Note that in the frameworks we have used here the learning rule is applied to Φ and only indirectly to W. In general Φ is not correlated to either or (see S1A, S1B and S1C Fig). The correlation only becomes visible after multiplying K and Φ (see Eq 15). Furthermore, note that we have excluded the reservoir weights Ω from the correlation because we model them as being static. For Efficient Coding, we have ignored changes to the fast weights (Ω) because they are just replicates of the regular weights W.

Manifold perturbations with dynamical latent variables

In Fig 2 we considered latent variables that were assigned values by some unspecified external control, and that once set the networks’ sole task was to retain them. In reality the functional role of most local circuits is doubtlessly more complex and likely involves manipulating the latent variables in relation to the inputs.

The static attractor model can easily be extended to allow for some computations on the latent variables. In particular, we can replace the decoder weights Φ with some other weights Γ and choose these so that the latent variables are modified before they are fed back into the network (Fig 4A). Notably, if choose (21) where , the dynamical variables will evolve according to (22)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Oscillating latent variables.

(A) By slightly modifying the architecture of Fig 1B, the network can be constructed to perform some computation on the latent variables, for example evolving them as a linear dynamical system (Eq 22). (B) The element-wise correlation between K and Γ for increasing frequencies f = ω/2π. (C) The element-wise correlation between synaptic weights before and after an inside-manifold perturbation (permuting the columns of K, solid lines), and before and after and outside-manifold perturbation (permuting the rows, dotted lines).

https://doi.org/10.1371/journal.pcbi.1007074.g004

In the Efficient coding framework, Eq 21 is explicitly applied to find the new weights Γ. In FORCE and the NEF however, the error function of the learning is changed so that Eq 21 will only be applied implicitly. Finding Γ using this implicit method has the benefit that Γ can represent both linear and non-linear transformations of x [14], thus enabling more complex dynamics than suggested by Eq 21.

Here we first consider the linear case of two-dimensional oscillators, i.e. (23) There are three reasons we study oscillating dynamics in lieu of some more complex dynamical system: (1) it is straight-forward to implement comparable dynamics in all three frameworks, (2) the frequency can be continuously varied to demonstrate the difference between fast and slow dynamics, and (3) periodic and quasi-periodic movements are often observed in natural behavior and weakly periodic neural activity patterns have been observed in certain brain regions e.g. motor cortex [21].

Following Fig 2C, 2F and 2I, we measured the correlation between the encoders K and the decoders Γ for different oscillation frequencies to study slow and fast dynamics of the neural modes. We found that the correlation between K and Γ decreased with increasing oscillator frequency. We expect this effect because from Eqs 23 and 21 we have (24) For the Efficient coding framework, Eq 21 is applied explicitly. Therefore, in Fig 4B the Efficient coding trace could also be viewed as the ideal correlation predicted by the theory. The NEF follows this closely, while the FORCE deviates a bit. In any case, seeing that Γ is correlated with K for low frequencies, we can conclude that even for dynamical latent-variables inside-manifold perturbations should require less change of the synaptic weights than outside-manifold perturbations. In Fig 4C we verify that this is indeed the case: for low frequencies the synaptic weights after an inside-manifold perturbation are highly correlated with the originals (solid lines) while the correlation is robustly (close to) 0 for outside-manifold perturbations. Thus, for low frequencies, only small changes of the synaptic weights are required to make an inside-manifold perturbation. The procedure for creating Fig 4B and 4C was very similar for creating figure Fig 3: the three networks consisted of N = 1000 neurons each, and inside and outside-manifold perturbations were given by Eqs 18 and 19, respectively.

Thus, we argue that introducing slow dynamics does not invalidate our claim that inside-manifold perturbations will be easier to learn. If the latent variables reflect motor behaviors, it is reasonable to assume that they vary with behavioral timescales (hundreds of milliseconds), i.e. less than 5-10 Hz. Indeed, the oscillations found by Churchland et al. [21] appears to be about 1 Hz. Note that oscillations of latent variables do not in general reflect oscillations of neural activity (δ, α, β, γ, etc.).

Networks with sparse and Daleian connectivity

While the networks created up to this point exhibit many biologically plausible features, the matrices of synaptic weights W are dense and do not obey Dale’s law [22]. Because our claim is centered around the similarity between these matrices before and after the perturbations, it is necessary to verify that adding biological constraints does not nullify the result. Therefore, we here introduce network sparsity and Dale’s law.

We focus in this section on the Neural Engineering Framework because its use of explicit optimization makes it relatively straight-forward to introduce constraints. However, in the NEF (Eq 8) the variable being optimized is the decoders Φ (or, in the non-stationary case, Γ), and not the synaptic weights W. Therefore, to directly impose constraints on W we changed the optimization procedure to (25) where (26) Thus, by appropriately choosing C, we can constrain W. We enforced sparsity and Dale’s law by setting (27) where R_ij is draw randomly from a uniform distribution between 0 and 1, ξ controls the sparsity and ϵ is the fraction of excitatory neurons. Also note that we have changed the target from x_target in Eq 8 to f(x_target) for some function f(⋅) in Eq 25, so that temporal dynamics can be introduced.

We used Eq 25 to construct a network with N = 5000 neurons and D = 4 latent variables. The increase in network size helps to prevent the in-degrees from being too small given the sparser connectivity. The dynamics f(⋅) were chosen so that the first two latent variables would be the leading and lagging components of a 2 Hz oscillator and the last two would similarly be the lead and lag of a 4 Hz oscillator. Additionally, we took advantage of NEF’s possibility of non-linearities and added a term to f(⋅) causing the oscillation amplitude to converge to 1 (see Methods).

We chose ϵ = 0.8, corresponding to 80% excitatory neurons and 20% inhibitory neurons and matching the observed ratio of excitatory and inhibitory neurons in the neocortex [23]. To achieve plausible sparsity we set ξ = 0.75 meaning only 25% of the connections were allowed. However, even when C_ij ≠ 0 it is still allowed for W_ij to be arbitrarily close to 0. Excluding connections numerically indistinguishable from 0, the connection probability was 10.8% for excitatory neurons (j ≤ 4000) and 19.0% for inhibitory neurons (j > 4000). These connection probabilities are well within the biological ranges [24–26].

The matrix W is shown in Fig 5A and exhibits many biologically plausible features in addition to being Daleian and sparse. For instance, the sum of all excitatory inputs matches the sum of all inhibitory inputs to every neuron (Fig 5C). Furthermore, the synaptic weights have a heavy-tail distribution (Fig 5D). Although a closer analysis reveals it to be somewhat leptokurtic (kurtosis is 4.56 for logarithmized weights), the weights distribution is in any case rather close to being log-normal across more than five orders of magnitude. This conforms with recent experimental data suggesting that the weight distributions in biological neural networks are typically heavy-tailed, if not perfectly log-normal [27].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The biologically plausible weights matrix W.

(A) The synaptic weight matrix is sparse and Daleian (neurons with id ≥ 4000 only make inhibitory connections). Only 100 excitatory and 25 inhibitory neurons are shown (B) Singular value decomposition of weight matrix W. W has more than four non-zero singular values, i.e. rank >D. (C) The sum of incoming excitatory weights to each neuron (one point) is similar to the sum of incoming inhibitory weights, indicating E/I balance. (D) Distribution of synaptic weights. The distribution is heavy-tailed but not perfectly log-normal. (E) Correlation between W and (inside manifold perturbation of K) and between W and (outside manifold perturbation). For comparison a third option with a completely independent K is also shown. Error bars indicate standard deviation over 23 different permutations.

https://doi.org/10.1371/journal.pcbi.1007074.g005

We then exposed this model to the same type of perturbations of the encoders K as we did for the previous models. However, while only one non-identity permutation matrix exists for D = 2, there are 4! − 1 = 23 non-identity permutations for D = 4. Still restricting our choice of perturbations to permutations, there are hence 23 possibilities for . We calculated the element-wise correlation between W and for all 23 perturbations. Similarly, we generalized Eq 19 to 4-fold permutations and calculated the element-wise correlation between W and for all 23 non-identity permutations. The correlations for inside-manifold perturbations were significantly higher than outside-manifold perturbations (Fig 5E, p = 6.1 ⋅ 10⁻⁷, dependent samples t-test). Note that the same C was used for all weight matrices and that this incurs some structure that increases the correlation. To assess the magnitude of this effect, we redrew 23 independent encoder matrices K′, computed the corresponding weight matrices and correlated them with W. These correlations were not significantly different than the outside-manifold perturbations (Fig 5E, p = 0.12, dependent samples t-test). In other words: although outside-manifold perturbations evince non-zero correlations, they still require changing the synaptic weights almost as much as when learning a completely new set of neural modes.

FORCE learning

We implemented the network described by Nicola and Clopath [12]. We normalized the units of Eq 2 so that (29) Setting the reverse potential at the threshold causes the neurons to be tonically active, removing the need of a driving current to start the network.

Following [12] we set the membrane time constant to τ_m = 10 ms, the synaptic time constant to τ_syn = 20 ms, and added a t_ref = 2 ms refractory time after each spike during which the membrane voltage was clamped to V_reset. Each element in the reservoir weight matrix Ω had a 90% probability of being set to 0, and is otherwise drawn from a normal distribution with zero mean and standard deviation (30) where G = 10 and N = 1000. In addition, the reservoir weights were balanced so that the average input weight to each neuron was 0.

The elements of encoder matrix K were drawn uniformly from [−100, 100] and the initial decoders Φ(0) where 0. The helper matrix P was initialized to (31)

The decoders and the the helper matrix were subsequently updated every 10th time-step when learning was active, using the following update rules: (32) (33) where e(t) = x(t) − x_target(t) is the error in the current decoding. The helper matrix is an online estimate of the inverse of the correlation matrix of u(t) (see [12]).

The network was simulated for a total of 50 seconds for each experiment, with time-step 0.05 ms. During the first 45 seconds, the network was repeatedly exposed to the 2.5 second long target pattern (the steps shown in Fig 2A and a sine and cosine for Fig 4) while learning (Eq 33) was active. During the last 5 seconds learning was not active and the last 2.5 seconds are shown in Fig 2A.

Learning was made more difficult in Fig 2 because of the instantaneous jumps in the target signal. To compensate for this, the network was first trained without the feedback connection. Instead, the target signal was used as input to the network. The feedback was then gradually reintroduced until the network was able independently keep the variables stable after the imposed jumps of latent variables. Note that because the illustration in Fig 2A is using the same latent variable values as was used during training, it is not an unbiased performance test.

Neural Engineering Framework

We used the NEF implementation in the simulator package Nengo [42] (version 2.6.0). We kept the default parameters of Nengo, but changed the synaptic time constant to τ_syn = 10 ms (membrane time constant was left at τ_m = 20 ms). Similarly, we used Nengo’s default way of choosing encoders K which is to draw each row uniformly from the unit sphere. However, we changed the max_rates parameter that controls the scaling of K so that the maximal firing rate of each neuron fell between 80 and 120 Hz. From the maximal firing rates the bias (I_drive in Eq 7) and the gain were determined automatically by Nengo for each neuron.

The simulation in Nengo is done in two steps. First, an estimated solution of Eq 8 is found and used to connect the network. Second, the network is simulated. To estimate the solution of Eq 8, Nengo draws a number of evaluation points from the space of allowed values of x_target. We used Nengo’s default space of allowed values, -1 to 1 for each latent variable, as well Nengo’s default heuristic to determine the number of evaluation points, which gave 2000 points. For each point, Nengo estimates what the expected filtered spike trains u(t) would be for that point, and then find the least-squares optimal solution to Eq 8 where the average is taken over the evaluation points.

To simulate fixation of variables during the shaded periods in Fig 2B, we connected the ensemble in Nengo to a node that was in turn connected back to the ensemble. The node was programmed to output the fixed signals during the shaded periods and to simply relay its input otherwise. In the non-shaded periods this is equivalent to recurrently connecting the ensemble to itself. We simulated the network for 2.5 seconds using the default Nengo time-step 1 ms.

Efficient coding

We implemented the network described by Boerlin et al. [15]. We normalized the membrane voltage so that V_leak = 0 and R_m = 1. The threshold was set individually for each neuron to (34) where ν = 10⁻³, μ = 10⁻⁶ and k_j is the jth row of K. No explicit reset is needed after a spike. Instead, the fast reservoir connections includes inhibitory autapses (from Eqs 10 and 11) which decrease V_m by after each spike. We used the same membrane time constant τ_m = 50 ms as [15], but chose a shorter synaptic time constant τ_syn = 20 ms ⇒ λ_syn = 50 Hz.

For Fig 2G, the latent variables actually used in the simulation were 100 times bigger than indicated on the axis. However, since the units of the latent variables are arbitrary we scaled it down in the plot to make comparison easier between the frameworks. To create the figure we simulated the network for 2.5 seconds with time-step 0.1 ms.

Customized learning rule for NEF

To demonstrate that our result can generalize to a biologically plausible weight matrix we modified Nengo to solve Eq 25 instead of Eq 8 during its building phase. We implemented this modification as a subclass of the Solver class in Nengo. Our subclass takes as input a matrix C and when called from Nengo, it uses the scipy routine nnls to solve the constrained optimization in Eq 25. Similarly, when solving Eq 8, the average in Eq 25 is estimated over a number of evaluation points. In the biologically plausible case, we explicitly increased the number of evaluation points to 40,000.

We did a few more changes to increase the biological plausibility compared to the previous simulations. First, we increased the network size to N = 5000 neurons and the number of latent variables to D = 4. We also chose a non-linear dynamical system for the latent variables to follow. Namely, we chose f(⋅) so that the first two encoded values would be the leading and lagging components of a 2 Hz oscillator and the last two would similarly be the lead and lag of a 4 Hz oscillator. Additionally, we added a non-linear term causing the amplitude to converge to 1. With these terms combined, the latent variables x(t) evolves as: (35) We used α = 0.2 which is sufficient to help stabilize the dynamics to oscillators with frequencies 2 and 4 Hz even in the absence of structured inputs. In the simulations shown in Fig 6 and S2 Fig, we initialized the latent variables x₁…x₄ to 0.

Simulation tools

Simulation and training of networks designed according to Liquid State Machine and Efficient Coding frameworks were performed using custom scripts written in Julia [43]. Networks designed according to the NEF were implemented using NENGO [42].

Measuring similarities

In Fig 2C, 2F and 2I, we want to measure the similarity between the subspaces spanned by each pair of matrices (K, Φ^T, Z_PCA and Z_PCA). To this end we measured the cosine of the mean principal angles (as computed by the scipy routine subspace_angles, see also [44]). This value ranges from 0 (unrelated, orthogonal space) to 1 (exactly the same space). This measure was also used in S2C Fig.

In S1A, S1B and S1C Fig on the other hand, we sought to demonstrate that the neural modes have indeed been permuted along with the encoders. This change is happening within the manifold/subspace, so angles between subspaces are no longer meaningful. Therefore, specifically for S1A, S1B and S1C Fig, we instead calculated the average of the absolutes of the correlations between the columns: (36)

Finally, for comparing weight matrices in Figs 3–5, we calculated the correlation over all pairs of old and new weights (i.e. element-wise). We have also reported the Frobenius norm of the difference in S1D and S2D Figs.