## Abstract

The study of the brain encompasses multiple scales, including temporal, spatial, and functional aspects. To integrate understanding across these different levels and modalities, it requires developing quantification methods and frameworks. Here, we present effective Geometric measures from Correlated Manifold Capacity theory (GCMC) for probing the functional structure in neural representations. We utilize a statistical physics approach to establish analytical connections between neural co-variabilities and downstream read-out efficiency. These effective geometric measures capture both stimulus-driven and behavior-driven structures in neural population activities, while extracting computationally-relevant information from neural data into intuitive and interpretable analysis descriptors. We apply GCMC to a diverse collection of datasets with different recording methods, various model organisms, and multiple task modalities. Specifically, we demonstrate that GCMC enables a wide range of multi-scale data analysis. This includes quantifying the spatial progression of encoding efficiency across brain regions, revealing the temporal dynamics of task-relevant manifold geometry in information processing, and characterizing variances as well as invariances in neural representations throughout learning. Lastly, the effective manifold geometric measures may be viewed as order parameters for phases related to computational efficiency, facilitating data-driven hypothesis generation and latent embedding.

The neural population doctrine [1, 2] postulates that populations of neurons serve as both functional and structural units in brain computation. While there have been an explosive amount of experimental findings supporting this idea [3, 4], we need to develop interpretable and computationally relevant quantifications of neural population activity. This way, we can generalize across multiple scales of analysis [5]. For example, we have developed receptive fields and tuning properties in single-unit analysis. What analogous concepts at the population level can facilitate conceptual reasoning and scientific hypotheses?

Researchers have applied numerous methodologies to address the aforementioned question. The *structural alignment* approach, which seeks to compare low-order statistics of neural activities [6, 7] has faced criticism for its failure to establish connections with downstream computations and its lack of descriptors at the population level [8, 9]. Meanwhile, the *functional alignment* approach, which selects brain models based on their similarity to specific task performance or predictivity [10, 11], struggles with the challenge of multiple models exhibiting highly similar performance. Lastly, the *dimensionality reduction* approach [12, 13], while addressing some limitations of the aforementioned methods, has garnered concerns on a number of characteristics: its unreliability (e.g., horseshoe effect [9, 14, 15]), the absence of a standardized approach for quantifying neural signatures, and its limited applicability to situations where neural activities are low-dimensional. Further, dimensionality reduction fails to adequately account for data showing a wide spectrum of structure, including high-dimensional structure in certain regions [3, 16, 17].

## Neural manifolds and manifold capacity theory

Neurons collectively represent task-relevant information in the brain [1, 2]. In neuroscience we informally call the collection of neural response vectors to some given task condition or input stimulus a “neural manifold” (see Fig. 1a for some examples) [18]. The neural manifold principle postulates that the geometrical, statistical, and structural properties of these manifolds are highly relevant to the study of the functional roles of population representations [18, 19, 20] (Fig 1.b).

The Manifold Capacity Theory (MCT) [28] quantifies the neural manifold’s representational efficiency through the classification capacity [29, 30], which measures the amount of linearly decodable information per neuron. Using tools from statistical physics and machine learning theory, the MCT analytically characterizes the classification capacity as a function of the shape of a manifold. Such an analytical formula for capacity further suggests the definition of computationally relevant geometric terms such as effective dimension and effective radius of neural manifolds. These effective geometric measures lead to applications in analyzing neural representations across biological datasets [31, 32, 33] and artificial neural networks [34, 35, 36].

However, the effective manifold geometric measures from MCT are ignorant to neural correlations, which play crucial roles in neural information processing [37]. Consequently, they have been limited to datasets with low correlations and sometimes yield inaccurate approximations to capacity [27]. This presents a fundamental research question: What effective geometric measures best capture capacity in the presence of neural correlations from biological datasets?

## Results

In this work, we introduce effective Geometric manifold measures from Correlated Manifold Capacity theory (GCMC). Our contributions are three-fold: (1) GCMC incorporates the complex correlation structure via new effective geometric measures, which explicate how manifold geometry influences downstream computational efficiency. (2) GCMC connects noise correlations to manifold geometry, hence it unifies the concept of correlations at different system scales. (3) GCMC enables multi-scale data analysis such as quantifying the spatial progression of encoding efficiency across brain regions (Fig. 3), revealing task-relevant temporal dynamics (Fig. 4), and characterizing the variances and invariances in learning (Fig. 5). We demonstrate the power and the applicability of GCMC in a wide spectrum of datasets [21, 22, 26, 38, 23, 24, 25, 39] with various task modalities (e.g., vision, perceptual decision, motor, spatial memory), various model organisms (e.g., mice, monkeys, humans, artificial neural networks), and various recording methods (e.g., electrophysiology, calcium imaging, fMRI) (Fig. 1a). Finally, the effective manifold geometric measures can be conceptualized as order parameters for phases associated with computational efficiency, aiding in the generation of data-driven hypotheses and latent embedding.

## Effective geometry for correlated neural manifolds

High-dimensional geometry can be deceptive, especially when it comes to correlations and computationally relevant geometry (see examples in Fig. SI1). To overcome this challenge, in GCMC, we derive effective manifold geometric measures that directly determine capacity based on a recent physics theory [27] for manifolds exhibiting correlations with each other (Fig. 1c, left). To be more specific, we derive from the theory the “anchor geometry space”, which intuitively is the effective geometry space seen by the optimal linear classifier (Fig. 1c, middle). From there, we extract effective geometric measures through isolating terms in the analytic formula for manifold capacity.

Six effective geometric measures are highlighted here: the effective radius, the effective dimension, the effective center norm, and three distinct forms of effective manifold alignments (as depicted in Fig. 1c, right). The analytical connection to the capacity formula says that an increase in “effective radius” and/or “effective dimension” leads to a decrease in manifold capacity. Intuitively, the effective radius and dimension respectively serve as high-dimensional generalizations of the inverse signal-to-noise ratio’s magnitude and degrees of freedom. The “effective center norm” captures the strength of mean neural responses whereas its effect on capacity has been factored into radius and dimension to ensure they are invariant to uniform scaling of the neural responses. A growth in “effective axes alignment” contributes to an increase in manifold capacity, as it allows for more efficient packing of manifolds oriented in the same direction within the neural state space. On the other hand, an increment in “effective center alignment” hurts manifold capacity, as it effectively reduces the available space for manifold packing by constraining manifolds to occupy a smaller portion of state space. Lastly, “effective center-axes alignment” provides insights into the extent of manifold variation distributed along the manifold center direction, which corresponds to the mean response. These effective alignment measures capture how the shared variability of a population of neurons affect downstream read-out efficiency. Note that these effective geometric measures are not independent to each other and they together orchestrate how manifold organization influences the capacity. More intuitions will be progressively developed through examples presented in the subsequent sections. In summary, effective radius, dimension, and center norm capture the geometric properties of individual manifolds while the three alignment measures delineate the effect of correlations between manifolds.

## Understanding different types of neural correlations through effective manifold geometric measures

Correlated variability among neurons intricately connects to information encoding and decoding performance [37, 40, 41] of neural representations. There are multiple factors, including interactions between neurons and correlations among the input stimuli, posing a challenge for mechanistic understanding: How do neuron-wise correlations affect the structures of information encoding at the populational level?

In GCMC, effective radius and dimension capture the inverse signal-to-noise ratio’s variability strength and degrees of freedom respectively. Furthermore, the three manifold alignment measures in GCMC further differentiate the effects of correlations from signal (i.e., manifold center or mean response) and intrinsic variation (i.e., invariances or noises). Here, we present an illustrative examples on neural noises. More examples are provided in Fig. SI4.

The direction of noise correlations with respect to the decision boundary (e.g., Fig. 2a,c) would affect the decoding accuracy and the amount of encoded information. While earlier research [37] quantifies the effects of noise correlations using information theory, GCMC is able to further unveil the diverse manners in which neurons exhibit correlations in terms of different manifold geometry (Fig. 2a-b). First, Fig. 2a and Fig. 2.b give two extreme examples of the manifold geometry of noise correlations. As real datasets are likely to encompass both types of noise correlations to certain degree, the effective geometric measures within GCMC offer a language to describe the computational consequences of neural noises at the population level. Second, GCMC can examine signals for multiple categories (instead of binary), and hence opens up avenues for analyzing complex datasets (e.g., multi-class, hierarchical classes).

### Quantifying the spatial progression of encoding efficiency across brain regions

The study of how neural representations progress across brain regions serves as one fundamental example. While neural tuning analysis has proven successful in early vision studies, higher visual areas present challenges due to the absence of clear single-neuron selectivity to higher-level concepts such as objects and faces. Consequently, prevailing analysis paradigms rely on data-driven frameworks like decoding analysis, comparison to representations from artificial neural network models, or visualization via dimensionality reduction. Although these methods excel in quantifying information in neural populations (e.g., decoding [42, 22]), ranking brain models [10], and providing intuitive insights (e.g., visualization [6, 43]), it remains a challenge to simultaneously capture the interaction between scales, e.g., read-out performance (top-down) and physiological structures (bottom-up).

GCMC excels at quantifying the stimulus-driven and/or task-driven structures in neural variabilities underlying behavioral invariance. Here, we utilize GCMC to give a unifying quantification of the spatial progression of encoding efficiency across various brain regions. These include electrophysiological recordings in monkey’s early visual ventral stream (Fig. 3a-b) and late visual ventral stream (Fig. 3c-d), and a human fMRI experiment featuring multiple regions of interest (see Fig. 3e-f). In all examples, we see a clear improvement of manifold capacity along the visual hierarchy. Moreover, the changes of underlying geometry are different at different stages and different stimuli structures. For example, the growth of capacity from V1 to V2 is driven mainly by the shrinkage of radius and center alignment (Fig. 3b). On the other hand, the growth of capacity from V4 to IT is simultaneously driven by the decrease of multiple factors, including radius, dimension, axes alignment, and center-axes alignment (Fig. 3d). Furthermore, the differences in effective manifold geometry across variations in stimuli reveal distinct organizational strategies of V4 and IT, warranting further study. In an fMRI dataset involving human subjects [26], we observed that the capacity difference between a target category (e.g., body parts or faces) and a control category (e.g., electronic devices) increases along the visual hierarchy. Notably, the greatest capacity gain occurs in brain regions known for their selectivity to the target category (e.g., the Extrastriate Body Area (EBA) exhibits the highest capacity gain for body part categories). While similar analyses using decoding methods (see Fig. SI9, Methods) did not reveal significant differences, our findings suggest that GCMC could be a valuable approach for investigating selectivity in brain regions. Further elaboration and additional details can be found in Methods and Fig. SI6, SI7, SI9.

### Revealing the temporal dynamics of task-relevant manifold geometry

In recent years, there have been exciting advancements in the neural population dynamics framework, encompassing both computational modeling and theoretical analysis [44, 4, 45]. A growing body of experimental evidence highlights the pivotal role of temporal dynamics in areas such as motor control, decision-making, and working memory, among others. Here, we demonstrate how GCMC can elucidate the connection between population dynamics and the associated computations.

We first apply GCMC to a monkey dataset on a delayed center-out reaching task with electrophysiological recordings from premotor area (PMd) and primary motor area (M1) of macaque monkeys [38] (Fig. 4a). We divide each trial into time bins of length 0.2 seconds and define target-specific neural manifolds as collections of population responses associated with the same reaching target and measure the manifold geometry. GCMC unveils a wealth of manifold temporal dynamics (Fig. 4b). For example, we observe a gradual increase in capacity during the delay period, followed by a sudden burst immediately after the onset of movement. Meanwhile, the radius and dimension exhibit the opposite pattern, agreeing with the fact that capacity is inversely related to them (Fig. 1c). There are also quantitative differences between PMd and M1, e.g., the dimension of PMd is higher than M1, which agree to the previous findings [38] using a heuristic definition based on a dimensionality reduction method. Previously, it is known that the neural activities during preparatory and movement stage lie in subspaces that are orthogonal to each other. Here, the temporal dynamics of manifold geometry, particularly the sudden burst of center-axes alignments and center norm during the brief interval between the go cue and movement onset, paint out a more comprehensive picture: the preparatory computation compresses the target-specific manifold along the center direction (as seen by the decrease of radius and dimension), whereas the movement computation pushes the manifolds outward along the center direction, effectively reducing the radius and dimension (Fig 4c).

Next, we investigate a monkey motion direction discrimination task regarding perceptual decision making (Fig. 4d). Specifically, we analyze an electrophysiological dataset with neural recordings from the prearcuate gyrus of macaque monkeys [23, 24]. Similarly, we divide trials into time bins of length 0.2 seconds and define neural manifolds as the collection of neural responses corresponding to the same outcome (left or right) and the same coherence level of random-dot motions. We first analyze the manifold geometry of two manifolds of the same coherence level. Consistent with task-performance, the manifold capacity is higher for the manifold pair with higher coherence level, and the capacity grows throughout the evidence-integrating period (Fig. 4e). We next analyze the manifold geometry of all the manifolds together. The three effective manifold alignments suggest the curve-shaped organization of decision manifolds (Fig. 4f), agreeing with previous findings in other brain areas related to decision making [46]. Furthermore, the difference in effective radius and dimension between manifolds with different coherence level indicates that there is a stronger compression when the monkey has more evidence (Fig. 4g).

### Characterizing variances and invariances in learning

To study the underlying mechanism of learning at the population level, it is essential to establish quantitative measures that can characterize the variances and invariances in neural representations [47]. The effective geometric measures in GCMC naturally serve as tools for quantifying the computationally relevant information in the presence of learning.

In a learning task with clear supervisions or rewards, we expect the animals (or artificial neural networks) to improve their neural presentations in a way that facilitates performance after training. Manifold capacity can be used to quantify how the read-out efficiency of neural representations improves over time, and hence GCMC further delineates the various “learning strategies” in terms of the changes of effective manifold geometry. For example, in a decision-making task in mice [39] (Fig. 5a), we find that the manifold capacity positively correlated with the behavioral performance and the improvement of capacity is mainly driven by the decrease of effective radius (Fig. 5b). Meanwhile, although not being strongly correlated with accuracy, there is a consistent decrease of effective dimension, center alignment, and center-axes alignment from early sessions to late sessions. On the other hand, for an object classification task trained on feedforward deep neural networks, we found that the increment of capacity across layers is mainly driven by the decrease of effective dimension (Fig. SI10).

When learning happens in a less straightforward or implicit manner, the effect on neural representations could be more complicated. For instance, the effective manifold geometry may exhibit unexpected trends or remain invariant, as demonstrated in the following two examples. Firstly, we analyze a hippocampal two-photon imaging dataset on mice spatial memory, where mice traversed linear tracks in morphed virtual reality environments [25] (Fig. 5c). In this study, mice experienced five different morph values with one group encountering all five environments throughout training (frequent morph), and the other group primarily encountering just two extreme morph environments (rare morph). The results were unanticipated: mice exposed to the rare morph condition exhibited higher capacity (across all five environments during the testing stage) compared to those exposed to the frequent morph condition (Fig. 5d). This potentially suggests that the rare morph condition facilitates formation of more compact hippocampal representations of the environment. In the second example, we investigated a rapid learning experiment involving a monkey reaching task [38], wherein monkeys received external perturbations after excelling at the task (see Fig. 5e). Our analysis reveals that the capacity of target-specific manifolds increased from the baseline after the introduction of external perturbations (Fig. 5f). Additionally, we observe variances as well as invariances in effective manifold alignment measures across various timescales (see Fig. 5h and Fig. SI5). Our results augment previous evidence suggesting that rapid motor adaptation is primarily due to a structural shift in the PMd population representation [38, 48].

## Discussion

Advances in science are tightly linked to the development of experimental tools and analytical methods. In the era of neural population doctrine, it is of great importance to develop analytical frameworks for mechanistically describing how populations of neurons encode complex task information. Here, we present effective Geometric measures derived from Correlated Manifold Capacity theory (GCMC) as quantitative descriptors for neural manifolds. These measures analytically link the neural population activities to the corresponding functional properties such as downstream read-out efficiency. They play an analogous role to neuronal tuning and hence can serve as data-analysis tools for investigating the temporal and spatial structures of neural populational representations. By applying it to a wide range of datasets, we have shown how GCMC can robustly reveal task-relevant information in different modalities, recording methods, modal organisms, tasks, etc. In the following, we will further elaborate on potential avenues for future research, discussing the new types of biological questions that can be tackled by GCMC and how to develop mechanistic understandings in neural networks.

GCMC differs from dimensionality reduction methods in three ways: (i) there is no information loss and the high-dimensional neural activities are summarized into interpretable geometric measures (Fig. 1b); (ii) these effective geometric measures are quantitatively related to capacity through the formula (see Methods) and the qualitative connection is described in Fig. 1c (right); (iii) the segregation of manifold center and within-manifold variability allows rich investigation into how correlations at different levels affect geometry and computational efficiency. Finally, GCMC offers a complementary approach to decoding methods [37, 42, 22], by distinguishing among representations which exhibit high similarity in decoding performance (Fig. SI1a, SI9, Methods).

GCMC facilitates at least three types of analysis at the population level: (i) establishing connections between representational geometry and normative principles, (ii) describing and quantifying neural computational strategies based on the dynamics of effective manifold geometry, and (iii) providing a unifying framework for the study of stimulus-driven and task-driven neural correlations.

Normative principles refer to top-down functional or computational objectives for a neural processing system, such as the efficient coding principle and the redundancy reduction hypothesis [49]. To test the validity of normative principles at the neural population level, we need quantitative measures for neural representations that can reflect downstream read-out properties. The effective geometric measures in GCMC, connected to the capacity of neural manifolds, offer a foundation for testing normative principles. For example, the decrease in effective radius and dimension (Fig. 3) supports the untangling hypothesis [42] along the visual hierarchy.

Neural computational strategies entail a mechanistic understanding of a neural system, with a focus on the algorithmic aspects of the system. For instance, in the monkey reaching task, researchers investigate how motor areas organize their neural population dynamics for preparation and movement [50]. Using dimensionality reduction methods, researchers have been able to describe populational neural computation in terms of “neural signatures” [51] and interactions across different “neural modes” [52]. With GCMC, one can not only enhance these mechanistic understandings by providing finer characterizations of the changes in manifold geometry (e.g., Fig. 4c,g), but also can offer a more detailed description of the underlying computational strategy in terms of the dynamics of effective manifold geometry.

Neural correlations are ubiquitous in the brain and are challenging to study due to the complex interactions among at least three factors: the stimuli, the neurons, and the tasks. By defining neural manifolds according to a class of stimuli (e.g., object and category manifolds in Fig. 3), a task outcome (e.g., direction manifolds in Fig. 4a-c), or both (e.g., outcome-coherence manifolds in Fig. 4d-g), GCMC offers a unifying framework to quantify the computationally relevant correlations in terms of effective representational geometry. Therefore, in future studies, our quantification method is flexible enough to investigate normative principles and/or neural computational strategies in the presence of neural correlations.

In a broader context, we propose that we regard the effective geometric measures for neural manifolds as “computational order parameters” for constructing models so as to gain insights into both biological and artificial neural networks. As a fundamental concept in statistical physics, an order parameter provides a quantitative description of phase transitions in complex systems. Furthermore, it often exhibits “universality”, meaning that different physical systems, regardless of their specific details and interactions, demonstrate similar critical behaviors near phase transitions and can be quantified using the same order parameter. Viewing effective manifold geometric measures as order parameters for a computational objective (i.e., capacity), GCMC allows the following two future directions (Fig. 6): (i) generating data-driven hypotheses and performing model selection via effective geometric measures, (ii) using effective manifold geometric measures as a latent embedding.

The formulation of scientific hypotheses relies on quantitative observables that can be extracted from the data and interpreted with respect to task behavior. For instance, neural tuning and selectivity represent fundamental quantification concepts in the context of the neuron doctrine. When it comes to the neural population doctrine, previous methodologies (e.g., similarity analysis, decoding performance, and geometric method) encounter at least one of the following two challenges: not formally connecting different levels of abstraction (e.g., structural and functional level) and being problem-specific. GCMC resolves these challenges by formally linking representational geometry to downstream read-out efficiency via the manifold capacity formula. By viewing effective manifold geometric measures as order parameters characterizing the computationally relevant properties of neural systems, GCMC provides an interpretable quantification for comparing neural systems. This enables a framework to perform hypothesis testing and model selection based on the closeness of effective manifold geometric measures [36].

A prominent approach in computational neuroscience and machine learning involves constructing latent models or embeddings [53] towards understanding a neural network. The concept is to embed neural representations into a low-dimensional and potentially interpretable latent space. Here, we propose using effective manifold geometry as a latent embedding. As the effective geometric measures from GCMC are order parameters for neural representations, using them as latent embedding offers several advantages. First, these measures quantify the computationally relevant properties of the system and are sample-efficient (due to the self-averaging property). Next, they are interpretable and hence can be used to develop conceptual understanding. If we further assume the universality of effective manifold geometric measures, then an advanced usage of GCMC would be building synthetic generative models and mapping them out in the effective manifold geometry space. We can then match a real neural network with a synthetic model via comparing their latent embeddings (i.e., distance in the manifold geometry space) (see Fig. 6a). It is worth noting that there are related approaches, such as approximate Bayesian computation [54] and simulation-based inference [55], which share the philosophy of inferring latent models through simulations or synthetic data. What sets our idea apart is the emphasis on making comparisons and selections at the “neural manifold” level through effective geometric measures of neural population activities. This provides a more interpretable and robust model comparison.

Lastly, it is an exciting direction for future research to apply GCMC to additional datasets or even design novel experimental setups suitable for neural manifold analysis. This approach opens up opportunities to explore new neuroscience questions at the neural population level.

## Data Availability

Monkey early vision dataset: Freeman et al. [21]. Monkey late vision dataset: Majaj et al. [22]. Monkey motor dataset: Perich et al. [38]. Monkey perceptual decision dataset: Kiani et al. [23, 24]. Human fMRI dataset: Hebart et al. [26]. Mice spatial memory dataset: Plitt et al. [25]. Mice decision-making dataset: Najafi et al. [39].

## Code Availability

Codes will be available public usage in the camera-ready version. Requests on accessing to the current version of the code should be made to the first and corresponding author.

## Author contributions

Conceptualization: CNC, AJW, SYC; Methodology: CNC, AJW, SYC; Software: CNC, LA, AJW; Data analysis and curation: CNC, LA, RK, WS; Writing-original draft: CNC; Writing-reviewing and editing: CNC, LA, AJW, RK, WS, SYC; Supervision: SYC; Funding acquisition: SYC.

## Conflicts

The authors declare no additional conflicts of interest. The funders had no role in the conceptualization, design, data collection, analysis, decision to publish, or preparation of the manuscript.

## Methods

### Datasets

#### Monkey early vision task

We use the dataset from Freeman et al. [21]. 13 anesthetized macaques were presented with a set of 450 images of 15 naturalistic textures while neural responses were recorded from visual cortex.

Each stimulus is an intact or scrambled version of one of 225 grayscale images. Each intact image is one of 15 randomly synthesized examples of a texture from one of 15 texture families. The images were scrambled by randomizing phase in the 2D Fourier domain, preserving a naturalistic power spectrum while destroying the high-order statistics distinguishing texture.

The stimuli were presented multiple times in pseudorandom order. Presentation lasted 100 ms with a 100 ms gap between presentations. Responses were recorded from several neurons at a time using quartz-platinum tungsten micro-electrodes (Thomas Recording) in cortical areas V1 and V2. Responses to 20 repeats of all stimuli were collected for 102 neurons in V1 and 103 neurons in V2.

#### Monkey late vision task

We use the dataset from Majaj et al. [22]. Two macaques were presented with images of objects rendered against naturalistic backgrounds while neural responses were recorded from visual cortex.

Each stimulus is one of 5760 grayscale images of a 3D object rendered against a randomly selected naturalistic background. The object is one of 64 3D models, where 8 example models belong to each of 8 object categories (animals, boats, cars, chairs, faces, fruits, planes, and tables). 90 stimuli are rendered from each 3D model by varying object pose, scale, location. Renderings are made at three levels of variation: none (10 images), medium (40 images), and high (40 images).

The stimuli were presented in series during passive fixation. Presentation lasted 100 ms with a 100 ms gap between presentations. Population responses were recorded using three 96-channel multi-electrode arrays in cortical areas V4 and IT. Responses to all stimuli were isolated for 189 sites in IT and 126 sites in V4 (a typical site captured spiking activity from 1-4 neurons).

#### Monkey reaching task

We use the dataset from Perich et al. [38]. Two macaques performed an eight-target center-out reaching task while neural responses were recorded from motor cortex.

A recording session consisted of about a hundred trials across three consecutive epochs: baseline, adaptation and washout. During baseline trials, the reaching task was not perturbed in any way. During adaptation trials, a motor perturbation was applied where either external forces were applied to the hand (“curl field”) or a fixed rotation was applied to the visual feedback (“visual rotation”).

Population responses were recorded using 96-channel multi-electrode arrays in cortical areas PMd and M1. Spike sorting was applied at the session level (9 curl-field and 7 visual-rotation sessions). Across all sessions, 137–256 PMd and 55–93 M1 neurons were isolated for Monkey C, and 66–121 PMd and 26–51 M1 neurons were isolated for Monkey M.

#### Monkey perceptual decision task

We use the dataset from Kiani et al. [23, 24]. Two macaques performed a random-dot direction discrimination task while neural responses were recorded from prearcuate cortex.

In each task trial, two targets appeared on the screen after the monkey directed its gaze towards a central point. After a 250 ms delay, a random dot motion stimulus was presented for 800 ms, followed by a variable-duration delay period. Upon presentation of a Go cue, the monkeys were instructed to indicate their perceived motion direction by executing a saccade toward the corresponding target. Motion strengths were tuned for each monkey to provide a broad range of behavioral accuracies, ranging from chance to flawless performance.

Population responses were recorded using 96-channel multi-electrode arrays in prearcuate gyrus. Spike sorting identified 100-250 units in each recording session (a unit could be an isolated single neuron or a multi-unit), for 15 sessions in total.

#### Mouse auditory decision-making

We use the dataset from Najafi et al. [39]. Four mice performed an auditory decision task while neural responses were imaged in posterior parietal cortex.

Trials consisted of simultaneous clicks and flashes, generated randomly (via a Poisson process) at rates of 5–27 Hz over 1,000 ms. Mice reported by licking a left or right water-spout whether event rates were high or low compared with a category boundary (16 Hz) learned from experience.

A two-channel, two-photon microscope was used to record the activity of neurons in layer 2/3 of posterior parietal cortex in transgenic mice. The images were processed by algorithm that simultaneously identified neurons and estimated *inferred spiking activity*, a number related to a neuron’s spiking activity during one frame. Recordings were made for 135 sessions in total, with a typical session isolating activity from 400-600 cells.

#### Human fMRI dataset

We use the dataset presented in Hbart et al. [26]. Three human participants were presented with images of objects while fMRI data were collected. The images were taken from the THINGS object concept and image database [56].

Each stimulus is one of 8640 colored images of an object embedded in a natural background. The object belongs to one of 720 distinct concepts, with each concept consisting of 12 exemplars.

The stimuli were presented in series while maintaining central fixation. Presentation lasted 500 ms with a 4000 ms gap between presentations. To maintain engagement, participants were instructed to perform an oddball detection task, detecting occasional artificially generated images amidst the stimulus presentation series.

Whole-brain fMRI data were collected using a 3 Tesla Siemens Magnetom Prisma scanner and a 32-channel head coil with 2mm isotropic resolution. Functional data was acquired for the main experiment for 12 scanning sessions and functional localizers were acquired for additional 1-2 scanning sessions. .

#### Artificial neural networks

We use a pretrained ResNet-50 architecture trained on ImageNet with learning algorithm SimCLR.

### Manifold capacity theory and GCMC

#### Mean-field model for correlated manifold capacity

Here we recall the mean-field model for correlated manifold capacity defined in [27]. Consider a collection of *P* manifolds *M*^{μ} lying in ℝ^{N} (e.g., *N* -dimensional neural state space or feature space), where *μ* = 1, 2, … *P* . These manifolds either come from the empirical data or a synthetic/mathematical model. As we focus on linear read-out, it suffices to consider the convex hull of each manifold. Thus, we describe manifold *M*^{μ} as where is the manifold center, forms the manifold intrinsic basis, and 𝒮^{μ} ⊂ ℝ^{K} describes the shape of the manifold (see Fig. 2 in [28] or Fig. 1a in [27] for pictorial examples). See later section (Code implementations) on how to measure these parameters from a real dataset.

The mean-field model for correlated manifold capacity considers the following problem. Let *κ* ≥ 0 be the margin. Let *y* ∈ {−1, 1}^{P} be a random binary label vector, and let be random Gaussian vectors in ℝ^{N} satisfying for some positive definite covariance tensor *C* (either measured from data or specified mathematically). The correlated manifold capacity *α*_{M} (*κ*) is defined as supremum of the ratio *P/N* in the thermodynamic limit (i.e., *P, N* → ∞) such that with probability 1 there exists a separating hyperplane *w* ∈ ℝ^{N} such that for every *μ*. Concretely, this can also be phrased as solving a log partition function as described in Equation 1 in [27].

The main result of [27] shows in its Equation 5 that
where *D*_{y,C} is the zero-mean Gaussian measure with covariance tensor , and the overline denotes the average over binary random label vector *y*. Next, 𝒜 is a convex set reflecting the geometry of the manifold shapes:
where *K* is the number of intrinsic coordinates of a manifold (in data analysis, *K* will be the number of points per manifold and for convenience we usually assume every manifold have the same *K*). Note that in the above equation, an element of 𝒜 is a matrix with rows and columns indexed by the superscript and the subscript of *V* respectively. Lastly, ∥· ∥_{y,C} denotes the Mahalanobis norm, i.e., Note that the above equation enables a simple algorithm to empirically calculate the correlated manifold capacity given the covariance tensor *C*: simply sample a bunch of Gaussians *T* and random labels *y* and take the empirical average of the left hand side of Equation 1.

#### The emergence of correlated anchor geometry

To extract manifold geometry from the capacity formula of [27], i.e., Equation 1, we first perform a change of basis to rotate the correlated Gaussian measure (i.e., *D*_{y,C}*T*) to the isotropic Gaussian measure as follows.
where now also reflects the correlations between manifolds as follows.
where *⊙* denotes the entry-wise multiplication (i.e., *z* = *x⊙y* where *z*_{i} = *x*_{i} *· y*_{i}) and
Next, we follow the idea from [28] to rewrite the term with “min” in Equation 3 by the KKT condition. Concretely, the minimizer *V* ^{∗} of min satisfies the following self-consistent equations:
As noticed in [28], the optimizer of the term min in Equation 8 is the anchor vector depending on *y* and *T* . Note that there might be multiple anchor points as the feasible set of the optimization problem might be large. Finally, by putting Equation 3 and Equation 8 together, we have
where the matrix *R* contains the anchor points *s*(*y, T*) on its rows, † denotes pseudo-inverse, and **1** denotes the all one vector. Note that as *R*^{⊤}(*RR*^{⊤})^{†}*R* is performing the orthogonal projection to the subspace spanned by the rows of *R*, the above equation is equivalent to Equation 1.

#### Effective manifold geometric measures from the capacity formula

Next, we again follow the intuition from [28] and decompose the anchor points *s*(*y, T*) into the longitudinal component (i.e., manifold center) *s*_{0} and the intrinsic component *s*_{1}(*y, T*), i.e., where *μ* is the manifold this anchor point corresponds to.

Concretely, is the mean of the anchor points coming from the *μ*-th manifold and .

With the above decomposition at hand, we can also decompose the matrix *R* = *R*_{0} + *R*_{1} where *R*_{0} contains on its rows and *R*_{1} contains *s*_{1}(*y, T*) on its rows. Thus, the gram matrix *RR*^{⊤} = *G*_{1} + *G*_{0} where we define and . Now, we are ready to define the effective manifold geometric measures for correlated manifolds. The key idea is using the Woodbury formula to rewrite (*G*_{1} + *G*_{0})^{†} and aggregating terms based on a high-dimensional analogy of what was done in [28]. For example, in [28], it is shown that the capacity *α* can be approximated by where *R*_{M} and *D*_{M} are effective manifold radius and dimension respectively.

Effective dimension:

Effective radius:

Effective center norm: where the overline over the inner term is over the anchor points correspond to the same Gaussian sampling (same notation in the following).

Effective axes alignment: where the superscript for

*s*_{1}(*y, T*)^{μ}indicates the index of the manifold that anchor point corresponds to.Effective center alignment:

Effective center-axes alignment:

Finally, we remark that one can potentially design more effective manifold geometric measures through the anchor point distribution.

#### Code implementations and analysis pipeline

The Python implementation of our effective manifold geometric measures is written in NumPy, SciPy, and CVXOPT, and a GitHub repository will be available in the camera-ready version of this manuscript.

To run the analysis, one simply construct a list `XtotT` of 2D arrays with the rows being neurons/units and the columns being points in the manifold, and execute `manifold_analysis_all(XtotT)` . The function will return a dictionary containing the resulting values of the effective manifold geometric measures. `manifold_analysis_all()` consists of several extra manifold analysis for comparison, one can use the argument mode to turn off those extra analyses (switching from ‘expensive’ to ‘cheap’). The main correlated geometry analysis lies in the function `manifold_analysis_corr()`, which contains the following three steps:

`prepare_manifold_data()`: empirically estimate the correlation tensor*C*(using Cholesky decomposition from SciPy) and turn the input data into the manifold shape space, i.e.,*S*^{μ}.`capacity_sampling()`: conduct Gaussian sampling based on Equation 3 and compute anchor points for each sample based on Equation 8 (using CVXOPT for solving a quadratic programming problem).`summarize_results()`: compute the correlated capacity based on Equation 1 and effective geometric measures based on Equation 10, 11, 12, 13, 14, 15.

### Synthetic examples

#### Example on neural noise correlations

For examples in Fig. 2 and Fig. SI3, we use the following generative model:

Parameters:

*N ∈*ℕ (ambient dimension);*P∈*ℕ (number of manifolds);*M ∈*ℕ (number of points per manifold);*ρ ∈*[0, 1) (positive noise correlations);*ψ ∈*[0, 1) (negative noise correlations).Procedures:

Randomly sample

*P*manifold centers*∈*ℝ^{N}independently from the G aussian distribution with mean 0 and variance for each coordinate.For each

*μ*= 1, …,*P*, randomly sample*M*vectors from the Gaussian distribution with mean 0 and variance for each coordinate.Let

*A*_{ρ}*∈*ℝ^{M ×M}be the square root of the matrix with diagonal being 1s and off-diagonal being*ρ*^{2}. Set the vectors to be the columns of*A*_{ρ}*U*where*U*contains in its columns.Define the

*μ*-th manifold to be the point cloud containing the following vectors:*·*for each*i=1,…,M*where

We pick *N* = 2000, *P* = 40, *M* = 40 in Fig. 2 and Fig. SI3 with 50 repetitions for each parameter setting. For case a in Fig. 2 and Fig. SI3, we set *ψ* = 0 and vary *ρ* from 0 to *0*.*8*. For case b in Fig. 2 and Fig. SI3, we set *ρ* = 0 and vary *ψ* 0 to 0.8.

#### Example on synthetic manifolds

For examples in Fig. SI2, we use the following generative model:

Parameters:

*N ∈*ℕ (ambient dimension);*P ∈*ℕ (number of manifolds);*D ∈*ℕ (manifold dimension);*M ∈*ℕ (number of points per manifold);*ρ*_{a}*∈*[0, 1) (axes correlations);*ρ*_{c}*∈*[0, 1) (center correlations);*ψ ∈*[0, 1) (center-axes correlations).Procedures:

Randomly sample

*P*manifold centers*∈*ℝ^{N}independently from the Gaussian distribution with mean 0 and variance for each coordinate.Let be the square root of the matrix with diagonal being 1s and off-diagonal being . Set the vectors to be the columns of where

*U*contains in its columns.For each

*μ*= 1, …,*P*, randomly sample*D*vectors from the Gaussian distribution with mean 0 and variance for each coordinate.Let be the square root of the matrix with diagonal being 1s and off-diagonal being . For each

*i*= 1, …,*D*, set the vectors to be the columns of where*U*contains in its columns.Define the

*μ*-th manifold to be the point cloud containing the following vectors: for each*j*= 1, …,*M*where*a*_{j}is independently sampled from the Gaussian distribution with mean 0 and variance*ψ*^{2}and is uniformly sampled from the unit sphere in ℝ^{D}.

We pick *N* = 1000, *P* = 30, *M* = 30, *D* = 20 in Fig. SI2 with 50 repetitions for each parameter setting. For each row in Fig. SI2a, we fix *ρ*_{a} *∈ {*0.0, 0.9*}, ψ* = 0, and vary *ρ*_{c} from 0.0 to 0.9. For each row in Fig. SI2b, we fix *ρ*_{c} *∈ {*0.0, 0.9*}, ψ* = 0, and vary *ρ*_{a} from 0.0 to 0.9. For each row in Fig. SI2c, we fix *ρ*_{a} *∈ {*0.0, 0.75*}, ρ*_{c} = 0, and vary *ψ* from 0.0 to 0.9. For each row in Fig. SI2b, we fix *ρ*_{c} *∈{*0.0, 0.75*}, ρ*_{a} = 0, and vary *ψ* from 0.0 to 0.9.

#### Example on manifolds with a 2D ring-shaped latent organization

The implementations of dimensionality reduction methods are based on SciPy and scikit-learn. The implementations of representational similarity analysis are based on the python package rsatoolbox. Here, we give a brief overview on the three synthetic manifolds models in Fig. 2. In all three cases, we generate 8 random spherical point clouds in a 200-dimensional space, each contains 20 points. Note that the choice of parameters is not fine-tuned and the results are very robust in a wide range of parameters. In Fig. 2a, we first generate a random Gaussian global center of expected length being 3.0. Next, we generate 2 random unit vectors and put 8 manifold centers on the unit circle spanned by these vectors with equal spacing. Finally, for each manifold, we generate 20 random Gaussian vectors of expected length 2.0 and add them to the manifold center to get the point cloud. In Fig. 2b, the generating process is exactly the same as that of the previous setting except in the final step we sample random points in a random 5-dimensional subspace (all the 8 manifolds use the same shared 5-dimensional subspace). In Fig. 2c, the generating process is exactly the same as that of the previous setting except the final step. For manifold 1,3,5,7, we generate 20 random Gaussian points of expected length 2.0 in the shared 5-dimensional subspace and a 20% bias toward the manifold center. For manifold 2,4,6,8, we generate 20 random Gaussian points of expected length 0.5. Lastly, in all the examples, we add very tiny random Gaussian noises (expected length being 0.01) to each point for numerical stability.

### Analysis

#### Standard form

The raw datasets span a variety of formats. We compile each dataset into “standard form”: a matrix whose columns are neural units and whose rows are population vectors sampled at varying task conditions and/or times throughout a trial. For vision datasets, we compile stimulus pixels into standard form as well.

#### Monkey early vision

The neural responses are a collection of population vectors corresponding to stimulus presentations. A population vector contains spike count in the 100 ms following response onset for each recorded neuron. In the pixel data, a population vector contains the intensity of each pixel in one image in the stimulus set.

We produce a set of manifolds for each population pixels, V1 and V2. For each population, we create 15 manifolds by grouping together neural responses to 15 different textures. Each manifold is a collection of presentation-averaged neural responses to a random subset of 10 images sampled from one texture. We randomly sample responses from a subset of 90 neurons in the ensemble. For pixel data, we use random projections to sample 90 dimensions from the 102400 pixel dimensions.

We assemble separate manifold sets from responses to scrambled and intact images. (We also compare against a random baseline where population responses are shuffled with respect to the stimuli before grouping into manifolds.)

We use GCMC to compute manifold capacity, radius, dimension, axes alignment, center alignment and center-axes alignment for each neural population. We show error bars across 10 random runs, and use a Mann-Whitney test to determine whether geometric measures differ significantly between neural populations.

#### Monkey late vision

The neural responses are population vectors corresponding to stimulus presentations. A population vector contains the z-scored number of spikes from 50 to 150 ms after stimulus onset at one electrode site.

We produce a manifold set for each population pixels, V4 and IT by grouping neural responses to images of the same 3D object. Each manifold is composed of responses from a random subset of 100 units to 30 randomly selected images of the object. For pixel data, we use random projections to sample 100 dimensions from the 196608 pixel dimensions.

We assemble separate manifold sets from images at all 3 levels of variation. For a random baseline, we shuffle neural responses with respect to the stimuli before grouping into manifolds.

We use GCMC to compute manifold capacity, radius, dimension, axes alignment, center alignment and center-axes alignment for each neural population. We aggregate results across 10 random runs and apply a Mann-Whitney test for statistically significant differences between populations.

#### Monkey motor task

The neural responses are spike rates from each neuron at varying time throughout a reaching task. Spike rates are derived by binning spike trains in 0.1 s intervals and convolving with a Gaussian kernel of band-width 0.2 s.

We produce manifold sets for each motor cortex population by grouping together neural responses which occur during the same reach direction, under the same trial block, and at the same time with respect to the planning period of the trial.

We run two analyses on the manifold geometry, one for studying the timecourse of neural responses across a trial and one for studying the effects of adaptation to the motor perturbation throughout a session.

In the first analysis, we produce manifolds by grouping together neural responses which occur during the same reach direction and at the same timebin throughout the course of a trial. We use a timebin of width 0.2 s with step size 0.1 s to sample manifold sets at varying times throughout the trial. At each timestep, we assemble direction manifolds by sampling 10 neural responses within the timebin for each of the 8 reach directions. We use responses from a random subsample of 50 units for each manifold set.

In the first analysis we compute the six manifold capacity metrics for PMd and M1 separately, and show each measured value as a function of time averaged across 5 random runs.

In the second analysis we create manifolds at the session level by sampling from neural responses occurring under the same reach direction and adaptation condition with respect to the motor perturbation. Correlations among these manifolds encode information about how the population representation changes structurally to adapt to perturbation of the task.

We delineate adaptation conditions by establishing “early” and ‘late” trial blocks within the “adaptation” and “washout” epoch as well as a “baseline” block of trials before the perturbation is induced. We define early and late blocks respectively as trials before and after the median trial index throughout the epoch, restricting analysis to those trials ending in reward.

We then compute GCMC correlation matrices encoding geometric alignments among the 40 reach direction-trial block manifolds. We run the analysis separately on manifolds sampled from before and after the onset of the reaching motion. We plot correlation matrices two ways by different ordering of the columns: first by assigning to adjacent columns manifolds from the same adaptation condition, and then by assigning to adjacent columns manifolds associated with the same reaching direction.

The main text shows analysis of the curl-field perturbation session on date 2016-09-15.

#### Monkey perceptual decision dataset

The neural responses are spike rates from neural units at varying time throughout a reaching task. Spike rates are derived by binning spike trains in 0.1 s intervals and convolving with a Gaussian kernel of bandwidth 0.2 s.

We produce manifold sets by grouping together neural responses which occur under the same dot-motion conditions and at the same timebin throughout the course of a trial. We use a timebin of width 0.2 s with step size 0.2 s to sample manifold sets at varying times throughout the trial. At each timestep, we assemble dot-motion manifolds by sampling neural responses within the timebin from trials with the same motion coherence and direction. Each of the 12 dot-motion manifolds (6 for coherence levels and two opposing directions) consists of 20 randomly sampled population vectors from the same subset of 50 neural units. We restrict analysis to trials where the perceptual decision was correct.

We compute manifold capacity metrics for each timebin separately, and show measured values as a function of time averaged across 10 random runs. Time ranges from -1.0 s to 3.0 s where 0 is dot stimulus onset. We also show the correlation matrices for manifolds from four select timebins throughout the evidence integration period of the trial (0.0, 1.0, 2.0 and 2.2 s).

#### Mice spatial memory dataset

The neural responses are the inferred spiking activities during each frame of ROIs extracted from calcium images. Each ROI correspond to individual neurons identified during postprocessing. The population responses are sampled at 30.9 Hz.

We produce manifold sets by grouping together neural responses during trials with the same stimulus condition: high- or low-frequency multisensory stimulation. Each manifold is 50 randomly sampled population vectors from all recorded neurons. Manifolds are constructed from responses in the 100 ms following the go tone of each trial.

Recordings span multiple days of each individual learning the task. We run GCMC analysis on manifolds for each recording session and compare across days. For each session, we aggregate manifold metrics across 10 random runs.

Then, for one individual, we plot each measured manifold metric as a function of task performance where the points are colored by the day on which that measurement was taken throughout the course of learning. The metrics show various relationships with task accuracy which generally increases across days.

#### Human fMRI dataset

The neural responses are single-trial response estimates of the BOLD response amplitude to each object image from each voxel. Single-trial estimates are derived by fitting a single-trial general linear model on the fMRI time series. Responses from 19 visual ROIs are used. (V1, V2, V3, hV4, VO1, VO2, TO1, TO2, V3b, V3a, EBA, FFA, OFA, STS, PPA, RSC, TOS, LOC, and IT).

We select three higher-level categories (face, body part, and electronic device) and produce a set of manifolds for each higher-level category and for each ROI. For each ROI, we create 4-5 manifolds by grouping together neural responses to 4-5 different concepts belonging to the same higher-level category. Each manifold is a collection of neural responses to a random subset of 10 images sampled from one concept. We randomly sample responses from a subset of 400 voxels.

We assemble separate manifold sets from responses to three higher-level categories - face, body part, and electronic device.

We use GCMC to compute manifold capacity, radius, dimension, axes alignment, center alignment, and center-axes alignment for each ROI and each higher-level category. We show error bars across 50 replicates and do a two-sample t-test for statistically significant differences between higher-level categories within each ROI.

#### Artificial neural networks

The neural responses are extracted from a pretrained ResNet-50 architecture trained on ImageNet with the SimCLR learning algorithm. We focus on 16 layers in ResNet-50: `x`, `conv1`, `layer1.0.relu`, `layer1.1.relu_2`, `layer2.0.relu_1`, `layer2.2.relu`, `layer2.3.relu_2`, `layer3.1.relu_1`, `layer3.3.relu`, `layer3.4.relu_2`, `layer4.0.relu_1`, `layer4.1.relu_1`, `layer4.2.relu`, `layer4.2.relu_1`, `layer4.2.relu_2`, `avgpool`. For each random repetition, we fix a random projection matrix for each layer and project the neural activations to a 2000 dimensional subspace. In each repetition, we randomly select 50 categories from the 1000 ImageNet categories and randomly select 30 images from the top 10% accurate images of each category to for the manifolds (we follow the same protocol as in ref. [34, 27]). We run GCMC analysis (as well as previous analysis methods for a comparison) on the manifolds of each layer and in total we conduct 5 random repetitions.

## Acknowledgements

We thank Matthew Perich for the insightful discussion on the monkey reaching dataset. We thank Roozbeh Kiani and Isabella Rischall for the helpful discussion related to monkey perceptual decision task. We also thank Billy Broderick, Jenelle Feather, Lada Kohoutová, Nikihil Parthasarathy, Won Mok Shim, and Choong-Wan Woo, for the stimulating conversation regarding many preliminary results. We thank Matthew Perich, Corey Ziemba, and Tony Movshon for share their datasets. This work was funded by the Center for Computational Neuroscience at the Flatiron Institute of the Simons Foundation and the Klingenstein-Simons Award to S.C. All experiments were performed on the Flatiron Institute high-performance computing cluster.

## Footnotes

↵✉ {cchou{at}flatironinstitute.org,schung{at}flatironinstitute.org}

## References

- [1].↵
- [2].↵
- [3].↵
- [4].↵
- [5].↵
- [6].↵
- [7].↵
- [8].↵
- [9].↵
- [10].↵
- [11].↵
- [12].↵
- [13].↵
- [14].↵
- [15].↵
- [16].↵
- [17].↵
- [18].↵
- [19].↵
- [20].↵
- [21].↵
- [22].↵
- [23].↵
- [24].↵
- [25].↵
- [26].↵
- [27].↵
- [28].↵
- [29].↵
- [30].↵
- [31].↵
- [32].↵
- [33].↵
- [34].↵
- [35].↵
- [36].↵
- [37].↵
- [38].↵
- [39].↵
- [40].↵
- [41].↵
- [42].↵
- [43].↵
- [44].↵
- [45].↵
- [46].↵
- [47].↵
- [48].↵
- [49].↵
- [50].↵
- [51].↵
- [52].↵
- [53].↵
- [54].↵
- [55].↵
- [56].↵
- [57].↵