ABSTRACT
The value of an integrated approach for understanding the neocortex by combining functional characterization of single neuron activity with the underlying circuit architecture has been understood since the dawn of modern neuroscience. However, in practice, anatomical connectivity and physiology have been studied mostly separately. Following in the footsteps of previous studies that have combined physiology and anatomy in the same tissue, here we present a unique functional connectomics dataset that contains calcium imaging of an estimated 75,000 neurons from primary visual cortex (VISp) and three higher visual areas (VISrl, VISal and VISlm), that were recorded while a mouse viewed natural movies and parametric stimuli. The functional data were co-registered with electron microscopy (EM) data of the same volume which were automatically segmented, reconstructing more than 200,000 cells (neuronal and non-neuronal) and 524 million synapses. Subsequent proofreading of some neurons in this volume yielded reconstructions that include complete dendritic trees as well the local and inter-areal axonal projections. The largest proofread excitatory axon reached a length of 19 mm and formed 1893 synapses, while the largest inhibitory axon formed 10081 synapses. Here we release this dataset as an open access resource to the scientific community including a set of analysis tools that allows easy data access, both programmatically and through a web user interface.
INTRODUCTION
Physiologists and anatomists have long been fascinated by the cerebral cortex, but aside from a few prominent examples, neuronal function and structure have been traditionally studied separately. In the 1970s, physiology and anatomy were combined to study single neurons in visual cortex by intracellular recordings of visual responses followed by dye filling and morphological reconstructions using light microscopy (Kelly and Van Essen 1974; Gilbert and Wiesel 1979). The goal of these experiments was to relate the receptive field of a neuron to its shape and anatomical location. This approach was also combined with electron microscopy (EM) to identify connectivity rules (E. L. White and Keller 1987; J. C. Anderson et al. 1994), but it typically yielded only one or a few cells per animal making it difficult to relate the structure and function of ensembles of neurons.
Since then, recording activity from large populations of single neurons has become possible using large scale electrophysiology and calcium imaging. Today the visual responses of tens or hundreds of thousands of neurons can be imaged simultaneously spanning multiple cortical areas in an awake behaving animal (Stringer et al. 2021; Demas et al. 2021; Walker et al. 2019). The “gold standard” in detailed anatomy, serial section electron microscopy, has also been scaled up to larger volumes, including an entire fly brain (Zheng et al. 2018) and a cubic millimeter of human temporal cortex (Shapson-Coe et al. 2021). By combining such technical advances, it should be possible to obtain a unified picture of physiology and anatomy for all cells within a modest volume of cortex.
Understanding cortical computations across layers and areas requires volumes that encompass entire dendritic arbors, entire local circuits and span distributed brain regions. In recognition of this opportunity, the IARPA Machine Intelligence from Cortical Networks (MICrONS) program launched in 2016 to coordinate the development of scalable technologies and generation of multimodal data sets that capture structure-function relationships at this scale.
Here we introduce the first large-scale MICrONS data resource, which is based on physiological and anatomical data acquired from approximately 1 mm3 (1.3×0.87×0.82 mm3 in vivo dimensions) of visual cortex in a P75-87 male mouse. This dataset is remarkable in a few key ways: It is the largest functional connectomics dataset released to date and the largest connectomics dataset by minimum dimension, number of cells, and number of synapses detected, and is 400× larger than a previously released pilot dataset from the MICrONS program. Due to its scale and location, the dataset contains multiple cortical visual areas enabling both intra- and inter-areal connectivity analysis. We will refer to these primary data and derived data products as the MICrONS multi-area dataset. The resource is composed of multiple data types (Fig 1) available online at www.microns-explorer.org.
Briefly, light and electron microscopic images were acquired from a cubic millimeter volume that spanned portions of primary visual cortex and three higher visual cortical areas. The volume was imaged in vivo by two-photon microscopy from postnatal days P75 to P81 in a male mouse expressing a genetically encoded calcium indicator in excitatory cells, while the mouse viewed natural movies and parametric stimuli. At P87 the same volume was imaged ex vivo by serial section EM. Because the light and electron microscopic images can be registered to each other, these primary data products in principle contain combined physiological and anatomical information for individual cells in the volume, with a coverage that is unprecedented in its completeness.
From the calcium images, we have extracted the responses of approximately 75,000 pyramidal cells to the visual stimuli using a constrained non-negative matrix factorization approach (Giovannucci et al. 2019). This includes the majority of the neurons with somas in the volume. From the EM images, cells and nuclei were automatically segmented using convolutional neural networks (K. Lee et al. 2017; Lu, Zlateski, and Sebastian Seung 2021; Wu et al. 2021). Segmented morphologies were used to automatically classify cells as neuronal versus non-neuronal and to further classify neurons into excitatory and inhibitory cell classes.
Already in the 1960s it was hypothesized that the receptive field or preferred stimulus of a neuron is highly dependent on its connectivity with other neurons (Hubel and Wiesel 1962). Our resource can be used to investigate such hypotheses, because synapses between neurons are visible in the EM images. In principle, it should be possible to extract a wiring diagram for the 75,000 pyramidal cells, and analyze it with respect to their visual responses.
Extracting this wiring diagram is a major challenge. Synapses between neurons were automatically detected and assigned to presynaptic and postsynaptic segments by convolutional neural networks. Because of the presence of segmentation errors, the automatically segmented cells need to be edited by human experts. This process of “proofreading is especially important for reconstructing accurate neural connectivity. However, the raw unproofread segmentation is typically sufficient for recognizing cells as belonging to a particular neuronal or non-neuronal class.
Strictly speaking, the assignment of synapses to presynaptic and postsynaptic cells yields a complete and accurate wiring diagram only for the neurons that have been fully proofread and are fully contained within the volume. However, for many questions, we can extract useful information about connectivity from neurons that have been proofread partially or even not at all, due to the high quality of the automatic dendritic segmentation (see Discussion).
Few studies combining calcium imaging and serial section EM exist, even if we expand to any brain region or species (Vishwanathan et al. 2017; Bae et al. 2018; Briggman, Helmstaedter, and Denk 2011; Wanner and Friedrich 2020). Previous applications of calcium imaging and serial section EM to cortex yielded wiring diagrams for tens (Bock et al. 2011; W.-C. A. Lee et al. 2016) or hundreds of cells (Dorkenwald et al. 2019; Schneider-Mizell et al. 2020; Nicholas L. Turner et al. 2020). Here we scale up to 75,000 cells with visual responses, and a wiring diagram that has the potential to grow similarly large as proofreading proceeds.
Data products
The primary in vivo data resource consists of two-photon calcium images, two-photon structural images, natural videos and parametric stimuli used as visual input, and behavioral measurements. The secondary (i.e. derived) in vivo data resource includes the responses of approximately 75,000 pyramidal cells from cortical layer 2 (L2) to 5 (L5) segmented from the calcium movies, pupil position and diameter extracted from the video of eye movements and locomotion measured on a single-axis treadmill.
The primary anatomical data are composed of ex vivo serial section transmission EM images spanning a volume roughly 1.3 mm (mediolateral) by 0.87 mm (anterior-posterior) by 0.82 mm (radial) when registered with the in vivo two photon structural stack. The volume includes a portion of primary visual cortex (VISp) and higher visual areas VISlm, VISrl, and VISal (Swanson 2018), for all cortical layers except extremes of L1. The serial section images were aligned to create a 3D image stack. The secondary anatomical data is derived from the serial section EM image stack, and consists of (a) automated segmentation of cells, (b) automated segmentation of nuclei, and (c) automatically detected synapses. The tertiary anatomical data consists of (a) assignments of the synapses to presynaptic and postsynaptic cells, (b) triangle meshes for these segments, (c) classification of nuclei as neuronal versus non-neuronal, and (d) classification of neurons into excitatory and inhibitory cell classes.
Secondary data for coregistration of in vivo and ex vivo images consists of manually chosen correspondence points between two-photon structural images and EM images. Tertiary coregistration data is a transformation derived from these correspondence points. The transformation is then used to facilitate the matching of cell indices between the two-photon calcium cell segmentation masks and the EM segmentation cells.
The public resource includes pyramidal neurons from all layers, such as L5 thick tufted, L5 NP, L4, and L2/3. The subvolume includes inhibitory neurons from many classes, such as bipolar cells, basket cells, a chandelier cell, and Martinotti cells. It also includes non-neuronal cells, such as astrocytes and microglia. The network of blood vessels is segmented, and contains some erroneously merged astrocytes.
Using the interactive visualization tools, one can visualize the input and output synapses of a single cell. The database of functional recordings is also available for download to explore how cells responded to visual stimuli. For a subset of the neurons, we have generated a correspondence between their functional recording and their anatomical reconstruction. The different data types are described in more detail below.
RESULTS
Overview
These data were collected from a single animal across a pipeline spanning three primary sites in the United States of America. First, two-photon in vivo calcium imaging under various visual stimulation conditions was performed at Baylor College of Medicine. Then the animal was shipped to the Allen Institute, where the imaged volume was extracted, prepared for EM imaging, sectioned and imaged over a period of 6 months of continuous imaging. The EM data were then montaged, roughly aligned and delivered to Princeton University, where fine alignment was performed and the volume was densely segmented. Finally, extensive proofreading was performed on a subset of neurons and various structural features were annotated (Fig. 2).
Functional Data
Neurophysiology Overview
The calcium imaging data includes the single-cell responses of an estimated 75,000 excitatory neurons to a variety of visual stimuli across multiple visual areas spanning cortical layers 2 through 5.
Two-Photon Calcium Imaging
Calcium imaging was performed from P75 to P81 in a transgenic mouse expressing GCaMP6s in excitatory neurons via SLC17a7-Cre (J. A. Harris et al. 2014) and Ai162 (Daigle et al. 2018). The calcium imaging dataset contains fourteen individual scans spanning a volume of approximately 1200 x 1100 x 500 µm3 (anteroposterior x mediolateral x radial depth, Fig 3a). The center of the volume was placed at the junction of primary visual cortex (VISp) and three higher visual areas, lateromedial area (VISlm), rostrolateral area (VISrl) and anterolateral area (VISal) in order to image retinotopically-matched neurons connected via long-range inter-areal feedforward and feedback connections.
Each scan was composed of multiple imaging planes, and leveraged the wide field-of-view of the two-photon random access mesoscope (2P-RAM) (Sofroniew et al. 2016) to capture two adjacent overlapping 620 µm-wide fields at each depth. The scans ranged up to approximately 500 µm in depth, with a target spacing of 10-15 µm to maximize the coverage of imaged cells in the volume (Fig 3b,c). For eleven of the fourteen scans, four imaging planes were distributed widely in depth using the mesoscope remote focus, spanning roughly 300-400 µm with an average spacing of approximately 125 µm between planes for near-simultaneous recording across multiple cortical layers. In the remaining three scans, fewer planes were imaged at 10-20 µm spacing to achieve a higher effective pixel density. These higher resolution scans were designed to be amenable to future efforts to extract signals from large apical dendrites from deeper layer 5 and layer 6 neurons, for example using EM-Assisted Source Extraction (EASE; Zhou et al. 2020). However, for this release, imaging data was automatically segmented only from somas using a constrained non-negative matrix factorization approach and fluorescence traces were extracted and deconvolved to yield activity traces (Giovannucci et al. 2019). In total, 125,413 masks were generated across fourteen scans, of which 115,372 were automatically classified as somatic masks by a trained classifier (Fig 3d, Giovannucci et al. 2019).
Behavioral Monitoring
During imaging, the animal was head-restrained, and the stimulus was presented to the left visual field. Treadmill rotation (single axis) and video of the animal’s left eye were captured throughout the scan, yielding the locomotion velocity, eye movements, and pupil diameter data included here.
Visual Stimulation
The stimulus for each scan was approximately 84 minutes, and consisted of natural and parametric movie stimuli. The majority of the stimulus (64 min) was made up of 10 second clips drawn from cinema, the Sports-1M (Karpathy et al. 2014) dataset, or rendered first-person POV movement through a virtual environment (Fig 3e). Our goal was to approximate natural statistical complexity to cover a sufficiently large feature space. These data can support multiple lines of investigation including applying deep-learning-based systems identification methods to build highly accurate models that predict neural responses to arbitrary visual stimuli (Sinz et al. 2018; Walker et al. 2019; Cotton, Sinz, and Tolias 2020). These models enable a systematic characterization of tuning functions with minimal assumptions compared to classical methods using parametric stimuli (Walker et al. 2019).
The stimulus composition included a mixture of unique stimuli for each scan, some that were repeated across every scan, and some that were repeated within each scan. In particular, six natural movie stimuli clips totalling one minute were repeated in the same order 10 times per scan, and were used to evaluate the reliability of the neural responses to repeated visual stimuli (Fig 3f). This “oracle score” serves as an important quality metric since reliable responses are not observed when imaging conditions are poor or the animal is not engaged with the stimulus To relate our findings to previous work, we also included a battery of parametric stimuli (“Monet2” and “Trippy”, 10 minutes each) that were generated to produce spatially decorrelated stimuli suitable for characterizing receptive fields, while also containing local or global directional and orientation components for extracting basic tuning properties such as orientation selectivity (Fig 3e,g).
Two-Photon Structural Stack
In order to densely target scan planes across multiple days, we needed a common reference frame to assess the coverage of scans within the volume. Therefore, in addition to the functional scans, a high-resolution structural volume was acquired for registration with the subsequent EM data. We collected two-photon structural stacks encompassing the functionally imaged volume at 0.5 - 1 px/µm resolution that served as a shared coordinate space. At the end of each imaging day, individual imaging fields of the functional scans were independently registered into the structural stacks with an affine transformation optimized via the pixel-wise correlation between the field and the stack (Fig 3b,c). This allowed us to target scans in subsequent sessions to optimize coverage across depth. On the last day of imaging, a two-channel (green, red) structural stack was collected at 0.5 px/um after injection of Dextran Texas Red fluorescent dye to label vasculature, enhancing fiducial labeling for co-registration with the EM volume (Fig 3a).
After registration of the functional imaging field with the structural stack, 2D centroids from the segmentation were assigned 3D centroids in the shared structural stack coordinate space. Using a greedy consolidation algorithm based on 3D proximity, we estimate that the functional imaging volume contains 75,909 unique functionally-imaged neurons consolidated from 115,372 segmented somatic masks, with many neurons imaged in two or more scans.
Electron Microscopy Imagery
After the in vivo neurophysiology data collection, we imaged the same volume of cortex ex vivo using Transmission Electron Microscopy (TEM) which allowed us to map the connectivity of neurons for which we measured functional properties.
Tissue sample
The volume of cortex prepared for histology contained the entire neurophysiology region of interest (ROI) dataset described above. This was confirmed by registering the in vivo two-photon structural stack with an X-ray microCT of the resin embedded tissue sample prior to sectioning.
The surface of the brain in the neurophysiology ROI was highly irregular, with depressions and elevations that made it impossible to trim all the resin from the surface of the cortex without removing layer 1 (L1) and some portions of layer 2 (L2). Though empty resin increases the number of folds in resulting sections, we left some resin so as to keep the upper layers (L1 and L2) intact to preserve inter-areal connectivity and the apical tufts of pyramidal neurons. Similarly, white matter was also maintained in the block to preserve inter-areal connections despite the risk of increased sectioning artifacts.
The tissue sample was originally composed of 27,972 serial sections at a nominal thickness of 40 nm that were collected onto grid tape (Phelps et al. 2021; Yin et al. 2020). The knife was cleaned every 100–500 sections, occasionally leading to the loss of a very thin partial section (≪ 40 nm). Thermal expansion of the block as sectioning resumed post-cleaning resulted in a short series of sections substantially thicker than the nominal cutting thickness. The sectioning took place in two sessions, the first session took 8 consecutive days on a 24/7 schedule and contained sections 1 to 14,773. The loss rate on this initial session was low, but before section 7931 there were two events that led to consecutive section loss. At the end of this session we started seeing differential compression between the resin and the surface of the cortex. Because this could lead to severe section artifacts, we paused to trim additional empty resin from the block. The second session lasted five consecutive days and an additional 13,199 sections were cut. Due to the interruption, block shape changes and knife replacement, there are approximately 45 partial sections at the start of this session; importantly, these do not represent tissue loss (see stitching and alignment section). As will be described later, the EM dataset is subdivided into two subvolumes due to sectioning and imaging events that resulted in loss of a series of sections.
Multiscope autoTEM imaging
In order to map the neurophysiology ROI to the ultrathin sections and define the region to be targeted for electron microscopy imaging (EM-ROI), low magnification images were taken on the transmission electron microscope and aligned with the microCT and 2P structural stack. A total of 26,652 sections were imaged by a fleet of five customized automated Transmission Electron Microscopes (autoTEMs, Yin et al. 2020), which took approximately 6 months to complete and produced a 2 PB dataset. Pixel sizes for all systems were calibrated within the range between 3.95∼4.05 nm/pix and the montages had a typical size of 1.2 mm × 0.82 mm. The EM dataset contains raw tile images with two different sizes because two cameras with two different resolutions were used during acquisition. The most commonly used was a 20MP camera that required 5,000 individual tiles to capture the 1 mm2 montage of each section. During the dataset acquisition, three autoTEMs were upgraded with 50MP camera sensors, which increased the frame size and reduced the total number of tiles required per montage to ∼2,600 (Fig 4d-h)
Volume assembly
The region between sections 7931 and 27904 (Fig 4a) was selected for further processing, as it had no consecutive section loss and an overall section loss of around 0.1%. This region contains approximately 95 million individual tiles that were further stitched into 2D montages per section and then aligned in 3D. Due to the re-trimming of the block and the knife change described above, the EM data are divided into two subvolumes (Fig 4a). One subvolume contains approximately 35% of the sections (section 7931 to 14815) and the other 65% containing sections 14816 to 27904. The two subvolumes were first stitched and aligned individually and later fine aligned to each other in the same global coordinate frame, allowing the tracing of axons and dendrites across their border (Fig 5). However, the two subvolumes were reconstructed separately and each has a distinct representation in the analysis infrastructure and database.
The sections were initially stitched using a per-image affine transformation model, and then a polynomial transformation model was applied to a subset of sections whose stitching quality with the affine model was above a local misalignment error threshold of 5 pixels. The 2D stitched sections were then roughly aligned in 3D using their downsampled versions. The rough alignment process ensures global consistency within the dataset and accounts for section images from multiple autoTEMs with varied image sizes and resolutions. It also corrects for locally varying misalignments such as scale differences and deformations between sections and aids the fine alignment process.
The alignment of the dataset was further refined using a set of convolutional networks to estimate displacement fields between pairs of neighboring sections. This process was able to correct misalignments around cracks and folds that occurred during sectioning. Though this fine alignment does not restore the missing data inside a fold, it was still effective in correcting the distortions caused by large folds (Fig. 4b-c), which are prevalent on this dataset and caused big displacements between sections; these were the main cause of reconstruction errors. To facilitate the reconstruction process across the division between the two subvolumes, a composite image of the partial sections was created.
Though the data was collected at ∼4 nm per pixel, the primary EM data that is being released has been downsampled to 8 nm per pixel, which was deemed sufficient for circuit reconstruction.
Segmentation of voxels and meshes
We segmented every cellular process in nearly the entire volume using affinity-predicting convolutional neural networks and mean-affinity agglomeration (K. Lee et al. 2017; Lu, Zlateski, and Sebastian Seung 2021; Wu et al. 2021; Dorkenwald et al. 2019). Segmentation was not attempted where the alignment accuracy was deemed insufficient or tissue was missing or occluded over multiple sections. For visualization and analysis, we provide triangular meshes (Fig 5a). The two subvolumes of the dataset were segmented separately, but the alignment between the two is sufficient for manually tracing between them (Fig 5b).
The automatic segmentation has highly accurate dendritic arbors, allowing for morphological identification of cell types. Most dendritic spines are properly associated with their dendritic trunk. Recovery of larger caliber axons, those of inhibitory neurons, and the initial portions of excitatory neurons was also typically successful. Due to the high frequency of imaging defects in the shallower and deeper portions of the dataset, processes near the pia and white matter often contain errors. Many non-neuronal objects are also well-segmented, including astrocytes, microglia, and blood vessels.
Synaptic connectivity
Synaptic clefts were automatically segmented or “detected” in the aligned EM image, and the presynaptic and postsynaptic partners from the cell segmentation were automatically assigned to identify each synapse (Fig 5d) using methods by Turner et al. (N. L. Turner et al. 2020; Wu et al. 2021). We automatically detected and associated a total of 524 million synaptic clefts across both subvolumes (subvolume 35: 186 million, subvolume 65: 337 million). We estimate that our cleft detection has a precision of 98.2% and recall of 98.2% using small volume test sets, while assignment accuracy was 98%.
Proofreading status
While the automated segmentation creates impressive reconstructions, proofreading is required to make those reconstructions more complete. The proofreading process involves both merging additional segments of the neurons that were missing in the reconstruction, and more accurate, by splitting segments that were wrongly associated with a neuron. To do this we used the ChunkedGraph proofreading system (Dorkenwald et al. 2020) which enables real-time collaborative proofreading in petascale datasets, and uses a modified version of Neuroglancer as a user interface.
The segmentation released here contains all of the proofreading that had occurred as of June 11, 2021 within the larger subvolume 65. This includes 601 neurons that have been extensively proofread, with 46,241 splits and 38,694 merges to correct errors. The level of completeness differs across these cells, as these neurons have been proofread as part of multiple MICrONS data analysis projects with different goals. For 78 excitatory neurons, both the dendrite and the axon were proofread to their maximal extension within the larger volume (incomplete ends were due to either the axon reaching the borders of the volume or an artifact that curtailed its continuation) (Fig 5c). These cells contain some of the most complete axonal arbors reconstructed at EM resolution. Axon arbor length ranges from 2.5 to 18.9 mm with a mean number of 714 synapses (range: 192-1893). For an additional 67 excitatory neurons we have only proofread the axon segments projecting from visual areas RL and AL to V1. For another 352 excitatory cells, only the dendritic arbor and soma have been proofread. Dendrite-only proofreading of a cell yields an accurate map of incoming dendritic and somatic synapses and provides valuable information for identifying cell type. Finally, we have also “cleaned” both the axons and dendrites of 104 inhibitory neurons, removing false merges but not extending all branches to their fullest extent. The number of output synapses on the proofread inhibitory cells ranges from 40 to 10051. In general, inhibitory cells are more complete in the automated reconstruction, likely because their axons are relatively thick. In addition to the manual proofreading, for the larger subvolume we also used the nucleus segmentation (see next section) to automatically detect and merge cases where nuclei were segmented separately from the surrounding cell.
Nucleus Segmentation and Cell Classification
We automatically detected 144,120 nuclei within the subvolume 65 after removing detections smaller than 25 μm3. We manually labeled a subset of the 2,751 nuclei in a 100 µm square column of the dataset as non-neuronal, excitatory or inhibitory. We then developed machine learning models to automate the detections of neurons (n=82,247), as well as excitatory (n=61,926) and inhibitory neurons (n=8,031) within the subvolume with high accuracy (see methods). The results of this nucleus segmentation, manual cell classification and model building are being made available as part of this data resource.
Functional - Structural Co-registration
Correspondence points and co-registration transform
Functional connectomics requires that cells are matched between the two-photon calcium imaging and EM coordinate frames. We manually matched 2934 fiducials between the EM volume and the two-photon structural dataset (1994 somata and 942 blood vessels, mostly branch points, which are available as part of the resource). Though the fiducials cover the total volume of the dataset it is worth noting that below 400 µm from the surface there is much lower signal to noise in the 2P structural dataset requiring more effort to identify somata, therefore we made use of more vascular fiducials. Using the fiducials, a transform between the EM dataset and the two-photon structural stack was calculated (see Methods). To evaluate the error of the transform we evaluated the distance in microns between the location of a fiducial after co-registration and its original location; a perfect co-registration would have residuals of zero microns). The average residual was 3.8 microns.
Cell Matching
Although the transform enabled good overall registration of the two volumes, the scientific cost of potential errors in the matching of individual cells across the two modalities motivated an extensive effort to manually validate many individual matches to yield high-confidence matches for analysis and provide “ground truth” for future fully-automated approaches.
We created a software infrastructure that allowed a group of trained “matchers” to use visual contextual features such as cell and vessel patterns from the 2P functional scans, 2P structural stack, and EM data to confirm matches with high confidence. Using this matching infrastructure we have included 200 manually-matched neurons in this release to demonstrate that matching is possible across the entire overlapping functional/EM volume.
Integrated Analysis
In order to create a resource for the neuroscience community, we have made the data from each of the steps described above — functional imaging, the EM subvolumes, segmentation, and a variety of annotations — publicly available on the MICrONS Explorer site (microns-explorer.org). From the site, users can browse through the large-scale EM imagery and segmentation results using Neuroglancer (J. Maitin-Shepard, https://github.com/google/neuroglancer); several example visualizations are provided to get started. All data is served from publicly readable cloud buckets hosted through Amazon AWS and Google Cloud Storage.
To enable systematic analysis without downloading hundreds of gigabytes of data, users can selectively access cloud-based data programmatically through a collection of open source Python clients (Fig 6a). The functional data, including calcium traces, stimuli, behavioral measures, and more, is available in a DataJoint (Yatsenko et al. 2015) database which can be accessed using DataJoint’s Python API (datajoint.org). EM imagery and segmentation volumes can be selectively accessed using cloud-volume (W. Silversmith, https://github.com/seung-lab/cloud-volume), a Python API that simplifies interacting with large scale image data. Mesh files describing the shape of cells can be downloaded with either cloud-volume or through MeshParty (S. Dorkenwald et al., https://github.com/sdorkenw/MeshParty), which also provides features for convenient mesh analysis and visualization. Annotations on the structural data, such as synapses and cell body locations, can be queried via CAVEclient, a Python interface to the Connectome Annotation Versioning Engine (CAVE) APIs (Fig 6a,b). CAVE encompasses a set of microservices for collaborative proofreading and analysis of large scale volumetric data. Note that, because CAVE services allow both uploading and downloading of information, using CAVE requires a simple, free registration. Instructions for bulk data downloading can also be found on Microns Explorer.
The first collection of annotation tables available through CAVEclient focus on the larger subvolume of the dataset, which we refer to within the infrastructure by the nickname “Minnie65”, and which has been the current focus of proofreading and ongoing analysis (Fig 6b). The largest table describes connectivity, containing all 337.3 million synapses and searchable by presynaptic id, postsynaptic id, and spatial location. In addition, there are several tables describing the soma location of key cells, predictions for which cells are excitatory and which are inhibitory (Fig 6c), cells that have been comprehensively proofread (Fig 6d), and a preliminary set of cells that have been matched to specific functional traces (Fig 6e). In this initial release, the only table available for the smaller subvolume (“Minnie35”) contains synapses, as its segmentation and alignment occured later and little proofreading, annotation or analysis has been conducted within it. We expect that continued proofreading and analysis of the data will lead to updated and additional tables for both portions of the data in future data releases.
This collection of tools and public data permits analyses that integrate questions of connectivity, morphology and functional properties of neurons. Here, we provide an example to suggest how the data might be used together. In order to seed an analysis with an as-complete-as-possible cell, you might begin by using the proofreading table to identify a neuron with complete axons and dendrites and querying for all the synaptic inputs and outputs for the cell, in this case a L2/3 cell in VISp (Fig 6e). Not all neurons are proofread, but we have empirically found that inputs to dendrites are often accurate so long as multiple cells are not merged together or disconnected from cell bodies. Here, we find that among the synaptic outputs of this proofread neuron, 54% (751/1398 synapses) are onto objects with a single nucleus (as determined from automated detection), with 246 synapses onto cells classified as inhibitory, 495 onto cells classified as excitatory, and 10 onto cells whose soma did not pass classification quality control (Fig 6g). Among the remainder, 16% (229/1398 synapses) are onto merged cells and 30% (418/1398) are onto orphan fragments. Both of these categories will be reduced over time as proofreading continues in the dataset. By filtering the synaptic targets with functionally-matched neurons (Fig 6h), you can further identify which targets have been matched to the functional experiments and use DataJoint to query the functional data (Fig 6h–l). In this case, the targets include pyramidal cells in both L2/3 and L5. Finally, the network can also be expanded with similar queries, for example finding all neurons that form intermediate feedforward connections between the presynaptic cell and the functionally matched targets (Fig 6m-n). Further use of the proofreading table can identify which intermediate cells have been proofread, assuring that the connections are not due to false merges. Here, the morphology of the intermediate cells establishes the intermediate cells as one L2/3 pyramidal cell and eight interneurons that arborize in layer 2/3 and layer 5. Subsequent investigation could look at the morphology of such cells in detail, or consider functional responses of their targets. We have provided example notebooks walking through the above examples and more to help users get started. Taken together, these data provide a platform for analysis of the relationship between the synaptic structure, neuronal morphology, and functional tuning of mouse visual circuits.
DISCUSSION
As electron microscopy is considered one of the gold standards to identify synapses, most datasets (ours included) were generated to address primarily questions of connectivity. However, the scale and high resolution of the MICrONS dataset offers information that is far richer and of broader interest than just connectivity. For example, the imagery also reveals the intracellular machinery of cells including the morphology of vital subcellular structures such as the nucleus, mitochondria, endoplasmic reticulum, and microtubules. Furthermore, the segmentation includes non-neuronal cells such as microglia, astrocytes, oligodendrocyte precursor cells and oligodendrocytes, as well the fine morphology of the cortical vasculature.
Comparison with other EM connectomics studies
The importance of high resolution structural data was recognized early in invertebrate systems, particularly in the worm with the classic studies of (Albertson, Thompson, and Brenner 1976; J. G. White et al. 1986) as well as more recent ones (e.g. (Bumbarger et al. 2013; Jarrell et al. 2012).
However, it is in the fly that connectomics, as the pursuit of complete connectivity diagrams, has had the strongest renaissance. For example, an EM volume containing the entire fly larva central nervous system was collected and released to the community (Ohyama et al. 2015). In addition, a whole central fly brain dataset which filled ∼33% of a 750 x 350 x 250 µm3 bounding box was produced and the imagery was also made available (Zheng et al. 2018). This volume contains ∼100,000 neurons and 244 million synapses (Buhmann et al. 2021). A smaller (250 x 250 x 250 µm3), but much more extensively proofread dataset from a portion of the fly central brain containing 25,000 cells and 20 million synapses is now available for analysis (Shan Xu et al. 2020). A dataset of the full adult fly nerve cord which occupies 35% of a 950 x 320 x 200 µm3 bounding box (Phelps et al. 2021) has also been produced. The creation of these datasets have spurred investment in both manual skeletonized reconstruction and automated dense reconstructions (Li et al. 2019; Dorkenwald et al. 2020) with related community minded efforts to proofread and mine them for biological insight (Scheffer et al. 2020; Dorkenwald et al. 2020; Schlegel et al., 2020; Fushiki et al. 2016; Bates et al. 2020; Marin et al. 2020). Taken together these datasets are beginning to cover the entire fly nervous system and are driving a revolution in how systems fly neuroscience is being studied.
In the mammalian system there is currently no EM dataset that contains a complete area, let alone a complete brain. There is however, as mentioned above, an established culture of making data open and publicly available (Kasthuri et al. 2015; J. R. Anderson et al. 2011; K. M. Harris et al. 2015). In the last 10 years there have been three other rodent EM-only datasets which are between 0.05% and 6% of the size of the MICrONS muli-area dataset presented in this manuscript, with publicly available reconstructions. One dataset is 424 × 429 × 274 µm3 from p26 rat entorhinal cortex (Schmidt et al. 2017) has skeleton reconstructions of incomplete dendrites of 667 neurons, and skeleton reconstructions of local axons of 22 excitatory neurons averaging 550 µm in length. A dataset from mouse LGN that is 500 x 400 x 280 µm3 is publicly available (Morgan et al. 2016), containing ∼3000 neuronal cell bodies and is large enough that dendritic reconstructions from the center of the volume are nearly complete. It has a sparse manual segmentation, covering ∼1% of the volume, that includes 304 thalamocortical cells and 162 axon fragments. The third is a 62 × 95 × 93 µm3 dataset from layer 4 of mouse somatosensory cortex contains 89 neurons and a dense automated segmentation with proofreading which attempts to cover the entire dataset of 153,171 synapses (Motta et al. 2019).
The most interesting neocortex is, of course, the human, and it is critically important to compare circuit architectures across species. There is already a large body of literature on the comparative aspects between the cortex of humans and of other species. This research includes morphological and electrical properties of neurons, density of spines, synapses and neurons, as well as biophysical properties and morphology of synaptic connections (DeFelipe, Alonso-Nanclares, and Arellano 2002; Mohan et al. 2015; Molnár et al. 2008, 2016; Testa-Silva et al. 2010; Yáñez et al. 2005; Elston, Benavides-Piccione, and DeFelipe 2001). Importantly, a recent EM connectomics dataset of the human medial temporal gyrus (Shapson-Coe et al. 2021) vastly expands the possibilities of this comparison. This is a mm3 scale volume, with a maximum extent of 3mm x 2mm and a thickness of 150µm. This human dataset is publicly available, including a dense automated reconstruction of all objects, with ∼16,000 neurons, 130 million synapses and an initial release of 104 proofread cells. These human connectomics data will doubtless yield critical insights. One practical difference from the volume described here is the aspect ratio of the human data, which is matched to the greater thickness of human cortex compared to mouse. To some extent the wide and thin dimensions of the human data trades off completeness of local neurons and circuits for breadth of connections, while the nearly cubic volume described here is more suitable for studying complete local circuits and long-range connections across areas.
None of the studies mentioned above have corresponding functional characterizations of the neurons reconstructed in EM. In contrast, the functional connectomics data we have released includes both anatomy and activity of the same cells.
The importance of functional connectomics
To mechanistically dissect the function of circuits, and visual circuits in particular, we must decipher their wiring and understand how this circuit structure relates to the functional properties of the constituent neurons. In 1972 Horace Barlow captured the importance of this level of detail in his “first dogma”:
A description of that activity of a single nerve cell which is transmitted to and influences other nerve cells, and of a nerve cell’s response to such influences from other cells, is a complete enough description for functional understanding of the nervous system. There is nothing else ‘looking at’ or controlling this activity, which must therefore provide a basis for understanding how the brain controls behaviour.
In invertebrate systems, specific neurons identified by morphology or genetics allow, in many cases, for structure and function to be studied in separate animals and integrated in downstream analysis. The diversity of functional responses and cell types in the neocortex makes this currently impractical and measuring both function and structure in the same neurons is thus critical. However, measuring the connectivity and function of specific neurons in the same animal is notoriously difficult for the obvious reason that functional properties like visual tuning must be measured in the intact animal in vivo, while connectivity at synaptic resolution is typically measured postmortem. Due to this technical difficulty there have been only a handful of datasets where synaptic-scale connectivity and functional responses were all collected in the same neurons, and those were constrained to small sample size. Some studies have related functional properties to connectivity measured via transsynaptic viral tracing (Rossi, Harris, and Carandini 2019; Wertz et al. 2015) co-registered multi-patching (Cossell et al. 2015; Hofer et al. 2011; Ko et al. 2011), but these have all been limited in the number and spatial distribution of cells and connections that could be explored.
The first functional connectomics studies which combined two photon microscopy and electron microscopy examined how the wiring of mouse retina (Vishwanathan et al. 2017; Bae et al. 2018; Helmstaedter et al. 2013; Briggman, Helmstaedter, and Denk 2011; Wanner and Friedrich 2020) and mouse visual cortex (Bock et al. 2011) related to functional properties. Lee and colleagues (W.-C. A. Lee et al. 2016) related visual tuning properties of 50 functionally-characterized neurons in primary visual cortex to their connectivity measured via EM reconstruction of a 450×450×150 um volume. The effort to reconstruct approximately 1000 of the synapses that these neurons made onto downstream cells was heroic. However, this effort yielded only 29 connected, functionally-characterized pairs. We previously released a 250 um x 140 um x 90 um dataset (Dorkenwald et al. 2019; Schneider-Mizell et al. 2020; Nicholas L. Turner et al. 2020) from mouse layer 2/3 visual cortex with a dense fully-automated reconstruction and manually proofreading of the dendrites and axons of the 398 neuronal cells contained within the volume. However, due to misalignment between the functional and structural data, only 112 excitatory cells have both functional data and reconstructed axons and dendrites, and so there are only 57 connected, functionally characterized pairs. In the olfactory bulb of the zebrafish, Wanner and colleagues (Wanner et al. 2016) manually reconstructed almost all neurons (n=1,003) within a 72×108×119 µm3 volume, where a 844 neuron subset had characterized responses to odors measured in vivo (Wanner and Friedrich 2020). Their analysis of the 18,483 measured connections revealed how this structural network mediated de-correlation and variance normalization of the functional responses and demonstrates how larger measurements of network structure and function can provide mechanistic insights.
In contrast, the data released here contains tens of thousands of neurons with functionally characterized responses to visual stimuli and because it is densely reconstructed, the opportunities to study connected neurons are orders of magnitude greater. As an example, from just 94 proofread excitatory axons, one can query 69,962 output synapses, which already map to 20,112 distinct neuron soma in the volume. We believe the opportunities for scientific discovery opened up by this leap in scale are vast. Many critical questions such as the relationship between connected groups of neurons, connections across layers, and functional and anatomical properties of feedback and feedforward connections simply cannot be answered by reconstruction of smaller volumes.
METHODS
Mouse Lines
All procedures were approved by the Institutional Animal Care and Use Committee (IACUC) of Baylor College of Medicine. All results described here are from a single male mouse, age 65 days at onset of experiments, expressing GCaMP6s in excitatory neurons via SLC17a7-Cre and Ai162 heterozygous transgenic lines (recommended and generously shared by Hongkui Zeng at Allen Institute for Brain Science; JAX stock 023527 and 031562, respectively). In order to select this animal, 31 (12 female, 19 male) GCaMP6-expressing animals underwent surgery as described below. Of these, 8 animals were chosen based on a variety of criteria including surgical success and animal recovery, the accessibility of lateral higher visual areas in the cranial window, the degree of vascular occlusion, and the success of cortical tissue block extraction and staining. Of these 8 animals, one was chosen for 40 nm slicing and EM imaging based on overall quality using these criteria.
Timeline
DOB: 12/19/17
Surgery: 2/21/18 (P64)
Two-photon imaging start: 3/4/18 (P75)
Two-photon imaging end: 3/9/18 (P80)
Structural Stack: 3/12/18 (P83)
Perfusion: 3/16/18 (P87)
Surgery
Anesthesia was induced with 3% isoflurane and maintained with 1.5 - 2% isoflurane during the surgical procedure. Mice were injected with 5-10 mg/kg ketoprofen subcutaneously at the start of the surgery. Anesthetized mice were placed in a stereotaxic head holder (Kopf Instruments) and their body temperature was maintained at 37 °C throughout the surgery using a homeothermic blanket system (Harvard Instruments). After shaving the scalp, bupivicane (0.05 mL, 0.5%, Marcaine) was applied subcutaneously, and after 10-20 minutes an approximately 1 cm2 area of skin was removed above the skull and the underlying fascia was scraped and removed. The wound margins were sealed with a thin layer of surgical glue (VetBond, 3M), and a 13 mm stainless-steel washer clamped in the headbar was attached with dental cement (Dentsply Grip Cement). At this point, the mouse was removed from the stereotax and the skull was held stationary on a small platform by means of the newly attached headbar. Using a surgical drill and HP 1/2 burr, a 4 mm diameter circular craniotomy was made centered on the border between primary visual cortex and lateromedial visual cortex (V1,LM; 3.5 mm lateral of the midline, ∼1mm anterior to the lambda suture), followed by a durotomy. The exposed cortex was washed with ACSF (25mM NaCl, 5mM KCl, 10mM Glucose, 10mM HEPES, 2mM CaCl2, 2mM MgSO4) with 0.3 mg/mL gentamicin sulfate (Aspen Veterinary Resources). The cortical window was then sealed with a 4 mm coverslip (Warner Instruments), using cyanoacrylate glue (VetBond). The mouse was allowed to recover for 1 day prior to imaging. After imaging, the washer was released from the headbar and the mouse was returned to the home cage. Prior to surgery and throughout the imaging period, mice were singly-housed and maintained on a reverse 12-hour light cycle (off at 11 am, on at 11 pm).
Two Photon Imaging
Mice were head-mounted above a cylindrical treadmill and calcium imaging was performed using Chameleon Ti-Sapphire laser (Coherent) tuned to 920 nm and a large field of view mesoscope (Sofroniew et al. 2016) equipped with a custom objective (excitation NA 0.6, collection NA 1.0, 21 mm focal length). Laser power after the objective was increased exponentially as a function of depth from the surface according to:
Here P is the laser power used at target depth z, P0 is the power used at the surface (not exceeding 10 mW), and Lz is the depth constant (not less than 150 μm.) Maximum laser output of 115 mW was used for scans approximately 450-500 μm from the surface and below.
Monitor Positioning
Visual stimuli were presented to the left eye with a 31.8 x 56.5 cm (height x width) monitor (ASUS PB258Q) with a resolution of 1080 x 1920 pixels positioned 15 cm away from the eye. When the monitor is centered on and perpendicular to the surface of the eye at the closest point, this corresponds to a visual angle of ∼3.8 °/cm at the nearest point and 0.7 °/cm at the most remote corner of the monitor. As the craniotomy coverslip placement during surgery and the resulting mouse positioning relative to the objective is optimized for imaging quality and stability, uncontrolled variance in animal skull position relative to the washer used for head-mounting was compensated with tailored monitor positioning on a six dimensional monitor arm. The pitch of the monitor was kept in the vertical position for all animals, while the roll was visually matched to the roll of the animal’s head beneath the headbar by the experimenter. In order to optimize the translational monitor position for centered visual cortex stimulation with respect to the imaging field of view, we used a dot stimulus with a bright background (maximum pixel intensity) and a single dark square dot (minimum pixel intensity). Dot locations were randomly ordered from a 5 x 8 grid to tile the screen, with 15 repetitions of 200 ms presentation at each location. The final monitor position for each animal was chosen in order to center the population receptive field of the scan field ROI on the monitor, with the yaw of the monitor visually matched to be perpendicular to and 15 cm from the nearest surface of the eye at that position. A L-bracket on a six dimensional arm was fitted to the corner of the monitor at this location and locked in position, so that the monitor could be returned to the chosen position between scans and across days.
Imaging Site Selection
The craniotomy window was leveled with regards to the objective with six degrees of freedom, five of which were locked between days to allow us to return to the same imaging site using the z-axis. Pixel-wise responses from a 3000 x 3000 μm ROI spanning the cortical window (150 μm from surface, five 600 x 3000 μm fields, 0.2 px/μm) to drifting bar stimuli were used to generate a sign map for delineating visual areas (Garrett et al. 2014). Our target imaging site was a 1200 x 1100 x 500 μm volume (anteroposterior x mediolateral x radial depth) spanning L2--L6 at the conjunction of lateral primary visual cortex (VISp) and three higher visual areas: lateromedial area (VISlm), rostrolateral area (VISrl), and anterolateral area (VISal). This resulted in an imaging volume that was roughly 50% VISp and 50% higher visual area (HVA). This target was chosen to maximize the number of visual areas within the reconstructed cortical volume, as well as maximizing the overlap in represented visual space. The imaging site was further optimized to minimize vascular occlusion and to minimize motion artifact, especially where the brain curves away from the skull/coverslip towards the lateral aspect.
Once the imaging volume was chosen, a second retinotopy mapping scan with the same stimulus was collected at 12.6 Hz and matching the imaging volume FOV with four 600 x 1100 μm fields per frame at 0.4 px/μm xy resolution to tile a 1200 x 1100 μm FOV at two depths (two planes per depth, with no overlap between coplanar fields). Area boundaries on the sign map were manually annotated.
Two Photon Functional Imaging
Of nineteen completed scans over 6 days of imaging, fourteen are described here. Scan placement targeted 10-15 μm increments in depth to maximize coverage of the volume in depth.
For 11 scans, imaging was performed at 6.3 Hz, collecting eight 620 x 1100 μm fields per frame at 0.4 px/μm xy resolution to tile a 1200 x 1100 μm FOV at four depths (two planes per depth, 40 μm overlap between coplanar fields).
For 2 scans, imaging was performed at 8.6 Hz, collecting six 620 x 1100 μm fields per frame at 0.4 px/μm xy resolution to tile a 1200 x 1100 μm FOV at three depths (two planes per depth, 40 μm overlap between coplanar fields).
For 1 scan, imaging was performed at 9.6 Hz, collecting four 620 x 1000 μm fields per frame at 0.6 px/μm xy resolution to tile a 1200 x 1000 μm FOV at two depths (two planes per depth, 40 μm overlap between coplanar fields).
In addition to locking the craniotomy window mount between days, the target imaging site was manually matched each day to preceding scans within several microns using structural features including horizontal blood vessels (which have a distinctive z-profile) and patterns of somata (identifiable by GCaMP6s exclusion as dark spots).
The full two photon imaging processing pipeline is available at (https://github.com/cajal/pipeline). Raster correction for bidirectional scanning phase row misalignment was performed by iterative greedy search at increasing resolution for the raster phase resulting in the maximum cross-correlation between odd and even rows. Motion correction for global tissue movement was performed by shifting each frame in X and Y to maximize the correlation between the cross-power spectra of a single scan frame and a template image, generated from the Gaussian-smoothed average of the Anscombe transform from the middle 2000 frames of the scan. Neurons were automatically segmented using constrained non-negative matrix factorization, then deconvolved to extract estimates of spiking activity, within the CAIMAN pipeline (Giovannucci et al. 2019). Cells were further selected by a classifier trained to separate somata versus artifacts based on segmented cell masks, resulting in exclusion of 8.1% of masks.
Two Photon Structural Stack
Approximately 55 minutes prior to collecting the stack, the animal was injected subcutaneously with 60 μL of 8.3 mM Dextran Texas Red fluorescent dye (Invitrogen, D3329). The stack was composed of 30 repeats of three 1300 x 620 μm fields per depth in two channels (green and red, respectively), tiling a 1300 x 1400 μm field of view (460 μm total overlap) at 335 depths from 21 μm above the surface to 649 μm below the surface. The green channel average image across repetitions for each field was enhanced with local contrast normalization using a gaussian filter to calculate the local pixel means and standard deviations. The resulting image was then gaussian smoothed and sharpened using a Laplacian filter. Enhanced and sharpened fields were independently stitched at each depth. The resulting stitched planes were independently horizontally and vertically aligned by maximizing the correlation of the cross-power spectrum of their Fourier transformations. Finally, the resulting alignment was detrended in Z using a Hann filter with a size of 60 μm to remove the influence of vessels passing through the fields. The resulting transform was applied to the original average images resulting in a structural two photon 1322 x 1412 x 670 μm volume at 0.5 x 0.5 x 0.5 px/μm resolution in both red and green channels.
Due to tissue deformation from day to day across such a wide field of view, some cells are recorded in more than one scan. To assure we count cells only once, we subsample our recorded cells based on proximity in 3-d space. Functional scan fields were independently registered using an affine transformation matrix with 9 parameters estimated via gradient ascent on the correlation between the sharpened average scanning plane and the extracted plane from the sharpened stack. Using the 3-d centroids of all segmented cells, we iteratively group the closest two cells from different scans until all pairs of cells are at least 10 μm apart or a further join produces an unrealistically tall mask (20 μm in z). Sequential registration of sections of each functional scan into the structural stack was performed to assess the level of drift in the z dimension. All scans had less than 10 μm drift over the 1.5 hour recording, and for most of them drift was limited to <5 μm.
Fields from the FOV matched retinotopy scan described above were registered into the stack using the same approach, and the manually annotated area masks were transformed into the stack. These area masks were extended vertically across all depths, and functional units inherit their area membership from their stack xy coordinates.
Eye and Face Camera
Movie of the animal’s eye and face was captured throughout the experiment. A hot mirror (Thorlabs FM02) positioned between the animal’s left eye and the stimulus monitor was used to reflect an IR image onto a camera (Genie Nano C1920M, Teledyne Dalsa) without obscuring the visual stimulus. An infrared 940 nm LED (Thorlabs M940L2) illuminated the right side of the animal, backlighting the silhouette of the face. The position of the mirror and camera were manually calibrated per session and focused on the pupil. Field of view was manually cropped for each session (ranging from 828 x 1217 pixels to 1080 x 1920 pixels at ∼20 Hz), such that the field of view contained the superior, frontal, and inferior portions of the facial silhouette as well as the left eye in its entirety. Frame times were time stamped in the behavioral clock for alignment to the stimulus and scan frame times. Video was compressed using Labview’s MJPEG codec with quality constant of 600 and stored the frames in AVI file.
Light diffusing from the laser during scanning through the pupil was used to capture pupil diameter and eye movements. Notably, scans using wide ranges in laser power to scan both superficial and deep planes resulted in a variable pupil intensity between frames. A custom semi-automated user interface in Python was built for dynamic adaptation of fitting parameters throughout the scan to maximize pupil tracking accuracy and coverage. The video is manually cropped to a rectangular region that includes the entirety of the eye at all time points. The video is further manually masked to exclude high intensity regions in the surrounding eyelids and fur. In cases where a whisker is present and occluding the pupil at some time points, a merge mask is drawn to bridge ROIs drawn on both sides of the whisker into a single ROI. For each frame, the original and filtered image is visible to the user. The filtered image is an exponentially-weighted temporal running average, which undergoes exponentiation, gaussian blur, automatic Otsu thresholding into a binary image, and finally pixel-wise erosion / dilation. In cases where only one ROI is present, the contour of the binary ROI is fit with an ellipse by minimizing least squares error, and for ellipses greater than the minimum contour length the xy center and major and minor radii are stored. In cases where more than one ROI is present, the tracking is automatically halted until the user either resolves the ambiguity, or the frame is not tracked (a NaN is stored). Processing parameters were under dynamic control of the user, with instructions to use the minimally sufficient parameters that result in reliably and continuous tracing of the pupil, as evidenced by plotting of the fitted ROI over the original image. Users could also return to previous points in the trace for re-tracking with modified processing parameters, as well as manually exclude periods of the trace in which insufficient reliable pupil boundary was visible for tracking.
Treadmill
The mouse was head-restrained during imaging but could walk on a treadmill. Rostro-caudal treadmill movement was measured using a rotary optical encoder (Accu-Coder 15T-01SF-2000NV1ROC-F03-S1) with a resolution of 8000 pulses per revolution, and was recorded at ∼57-100 Hz in order to extract locomotion velocity.
Stimulus Composition
Each scan stimulus is approximately 84 minutes and comprised:
Oracle Natural Movies: 6 natural movie clips, 2 from each category. 10 seconds each, 10 repeats per scan, 10 minutes total. Conserved across all scans.
Unique Natural Movies: 144 natural movies, 48 from each category. 10 seconds each, 1 repeat per scan, 24 minutes total. Unique to each scan.
2x Repeat Natural Movies: 90 natural movies, 30 from each category. 10 seconds each, 2 repeats (one in each half of the scan), 30 minutes total. Conserved across all scans.
Local Directional Parametric Stimulus (“Trippy”): 20 seeds, 15 seconds each, 2 repeats (one in each half of the scan), 10 minutes total. 10 seeds conserved across all scans, 10 unique to each scan.
Global Directional Parametric Stimulus (“Monet”): 20 seeds, 15 seconds each, 2 repeats (one in each half of the scan), 10 minutes total. 10 seeds conserved across all scans, 10 unique to each scan.
Each scan is also preceded by 0.15 - 5.5 minutes with the monitor on, and followed by 8.3 - 21.2 minutes with the monitor off, in order to collect spontaneous neural activity.
Natural Visual Stimulus
The visual stimulus was composed of dynamic stimuli, primarily including natural video but also including generated parametric stimuli with strong local or global directional component. Natural video clips were 10 second clips from one of three categories:
Cinematic from the following sources -- Mad Max: Fury Road (2015), Star Wars: Episode VII - The Force Awakens (2015), The Matrix (1999), The Matrix Reloaded (2003), The Matrix Revolutions (2003), Koyaanisqatsi: Life Out of Balance (1982), Powaqqatsi: Life in Transformation (1988), Naqoyqatsi: Life as War (2002).
Sports-1M collection (Karpathy et al. 2014) with the following keywords: cycling, mountain unicycling, bicycle, bmx, cyclo-cross, cross-country cycling, road bicycle racing, downhill mountain biking, freeride, dirt jumping, slopestyle, skiing, skijoring, alpine skiing, freestyle skiing, greco-roman wrestling, luge, canyoning, adventure racing, streetluge, riverboarding, snowboarding, mountainboarding, aggressive inline skating, carting, freestyle motocross, f1 powerboat racing, basketball, base jumping.
Rendered 3D video of first person POV random exploration of a virtual environment with moving objects, produced in a customized version of Unreal Engine 4 with modifications that enable precise control and logging of frame timing and camera positions to ensure repeatability across multiple rendering runs. Environments and assets were purchased from Unreal Engine Marketplace. Assets chosen for diversity of appearance were translated along a piecewise linear trajectory, and rotated with a piece-wise constant angular velocity. Intervals between change points were drawn from a uniform distribution from 1 to 5 seconds. If a moving object encountered an environmental object, it bounced off and continued along a linear trajectory reflected across the surface normal. The first person POV camera followed the same trajectory process as the moving objects. Light sources were the default for the environment. Latent variable images were generated by re-generating the scenes and trajectories, rendering different properties, including: absolute depth, object identification number, and surface normals.
All natural movies were temporally resampled to 30 fps, and were converted to grayscale with 256 x 144 pixel resolution with FFmpeg (ibx264 at YUV4:2:0 8bit). Stimuli were automatically filtered for upper 50th percentile Lucas-Kanade optical flow and temporal contrast of the central region of each clip. All natural movies included in these experiments were further manually screened for unsuitable characteristics (ex. fragments of rendered videos in which the first person perspective would enter a corner and become “trapped” or follow an unnatural camera trajectory, or fragments of cinematic or Sports-1M containing screen text or other post-processing editing).
Global Directional Parametric Stimulus (“Monet”)
To probe neuronal tuning to orientation and direction of motion, a visual stimulus was designed in the form of smoothened Gaussian noise with coherent orientation and motion. Briefly, an independently identically distributed (i.i.d.) Gaussian noise movie was passed through a temporal low-pass Hamming filter (4 Hz) and a 2-d Gaussian spatial filter (σ = 3.0° at the nearest point on the monitor to the mouse). Each 15-second block consisted of 16 equal periods of motion along one of 16 unique directions of motion between 0-360 degrees with a velocity of 42.8 degrees/s at the nearest point on the monitor. The movie was spatial filtered to introduce a spatial orientation bias perpendicular to the direction of movement by applying a bandpass Hanning filter G(ω; c) in the polar coordinates in the frequency domain for ω = ϕ − θ where ϕ is the polar angle coordinate and θ is the movement direction θ. Then:
Here, c=2.5 is an orientation selectivity coefficient. At this value, the resulting orientation kernel’s size is 72° full width at half maximum in spatial coordinates.
Local Directional Parametric Stimulus (“Trippy”)
To probe the tuning of neurons to local spatial features including orientation, direction, spatial and temporal frequency, the “Trippy” stimulus was synthesized by applying the cosine function to a smoothened noise movie. Briefly, a phase movie was generated as an independently identically distributed (i.i.d.) uniform noise movie with 4 Hz temporal bandwidth. The movie was upsampled to 60 Hz with the Hanning temporal kernel. An increasing trend of 8π/s was added to the movie to produce drifting grating movements whereas the noise component added local variations of the spatial features. The movie was spatially upsampled to the full screen with a 2D Gaussian kernel with a sigma of 5.97 cm or 22.5° at the nearest point. The resulting stimulus yielded the local phase movie of the gratings, from which all visual features are derived analytically.
Stimulus Alignment
A photodiode (TAOS TSL253) was sealed to the top left corner of the monitor, where stimulus sequence information was encoded in a 3 level signal according to the binary encoding of the flip number assigned in-order. This signal was recorded at 10 MHz on the behavior clock (MasterClock PCIe-OSC-HSO-2 card). The signal underwent a sine convolution, allowing for local peak detection to recover the binary signal. The encoded binary signal was reconstructed for 89% of trials. A linear fit was applied to the trial timestamps in the behavioral and stimulus clocks, and the offset of that fit was applied to the data to align the two clocks, allowing linear interpolation between them.
Oracle Score
We used six natural movie conditions that were present in all scans and repeated 10 times per scan to calculate an “Oracle score” representing the reliability of the trace response to repeated visual stimuli. This score was computed as the jackknife mean of correlations between the leave-one-out mean across repeated stimuli with the remaining trial.
Manual Matching
A custom user interface was used to visualize images from both the functional data and EM data side-by-side to manually associate functional units to their matching EM cell counterpart and vice versa. To visualize the functional scans, summary images were generated by averaging the scan over time (average image) and correlating pixels with neighbor pixels over time (correlation image). The product of the average and correlation images were used to clearly visualize cell body locations. Using the per field affine registration into the stack, a representative image of labeled vasculature corresponding to the registered field was extracted from the stack red channel. EM imagery and EM nucleus segmentation was resized to 1 um3 resolution, and transformed into the two-photon structural stack coordinates using the coregistration, allowing an image corresponding to the registered field to be extracted. The overlay of the extracted vessel field and extracted EM image were used to confirm local alignment of the vasculature visible in both domains. Soma identity was assessed by comparing the spatial structure of the target soma and nearby somas in the functional image to soma locations from the EM cell nuclei image. Using the tool, matchers generated a hypothesis for which EM cell nucleus matched to a given functional unit or vice versa. A custom version of Neuroglancer (Seung lab, https://github.com/seung-lab/neuroglancer) was used to visualize the region of interest in the ground-truth EM data for match confirmation. To choose the 200 matching cells for this release, the boundaries of the overlapping volume were determined by the intersection of the two bounding boxes generated from the functional unit centroids and transformed EM centroids, and then trimmed to 400 x 1000 x 500 um3 (anteroposterior x mediolateral x radial depth) to evenly divide into 200, 100 um3 bins. Candidate EM cells in each bin with relatively large and clean meshes were prioritized for matching.
Tissue Preparation
After optical imaging at Baylor College of Medicine, candidate mice were shipped via overnight air freight to the Allen Institute. All procedures were carried out in accordance with the Institutional Animal Care and Use Committee at the Allen Institute for Brain Science. All mice were housed in individually ventilated cages, 20-26 C, 30-70% Relative Humidity, with a 12-hour light/dark cycle. Mice were transcardially perfused with a fixative mixture of 2.5% paraformaldehyde, 1.25% glutaraldehyde, and 2 mM calcium chloride, in 0.08 M sodium cacodylate buffer, pH 7.4. After dissection, the neurophysiological recording site was identified by mapping the brain surface vasculature. A thick (1200 um) slice was cut with a vibratome and post-fixed in perfusate solution for 12 – 48 h. Slices were extensively washed and prepared for reduced osmium treatment (rOTO) based on the protocol of Hua and colleagues (Hua, Laserstein, and Helmstaedter 2015). All steps were performed at room temperature, unless indicated otherwise. 2% osmium tetroxide (78 mM) with 8% v/v formamide (1.77 M) in 0.1 M sodium cacodylate buffer, pH 7.4, for 180 minutes, was the first osmication step. Potassium ferricyanide 2.5% (76 mM) in 0.1 M sodium cacodylate, 90 minutes, was then used to reduce the osmium. The second osmium step was at a concentration of 2% in 0.1 M sodium cacodylate, for 150 minutes. Samples were washed with water, then immersed in thiocarbohydrazide (TCH) for further intensification of the staining (1% TCH (94 mM) in water, 40 °C, for 50 minutes). After washing with water, samples were immersed in a third osmium immersion of 2% in water for 90 minutes. After extensive washing in water, lead aspartate (Walton’s (20 mM lead nitrate in 30 mM aspartate buffer, pH 5.5), 50 °C, 120 minutes) was used to enhance contrast. After two rounds of water wash steps, samples proceeded through a graded ethanol dehydration series (50%, 70%, 90% w/v in water, 30 minutes each at 4 °C, then 3 x 100%, 30 minutes each at room temperature). Two rounds of 100% acetonitrile (30 minutes each) served as a transitional solvent step before proceeding to epoxy resin (EMS Hard Plus). A progressive resin infiltration series (1:2 resin:acetonitrile (e.g. 33% v/v), 1:1 resin:acetonitrile (50% v/v), 2:1 resin acetonitrile (66% v/v), then 2 x 100% resin, each step for 24 hours or more, on a gyrotary shaker) was done before final embedding in 100% resin in small coffin molds. Epoxy was cured at 60 °C for 96 hours before unmolding and mounting on microtome sample stubs.
The sections were then collected at a nominal thickness of 40 nm using a modified ATUMtome (RMC/Boeckeler, Yin et al. 2020) onto 6 reels of grid tape(Phelps et al. 2021; Yin et al. 2020).
Transmission Electron Microscopy Imaging
The parallel imaging pipeline described here (Yin et al. 2020) converts a fleet of transmission electron microscopes into high-throughput automated image systems capable of 24/7 continuous operation. It is built upon a standard JEOL 1200EXII 120kV TEM that has been modified with customized hardware and software. The key hardware modifications include an extended column and a custom electron-sensitive scintillator. A single large-format CMOS camera outfitted with a low distortion lens is used to grab image frames at an average speed of 100 ms. The autoTEM is also equipped with a nano-positioning sample stage that offers fast, high-fidelity montaging of large tissue sections and an advanced reel-to-reel tape translation system that accurately locates each section using index barcodes for random access on the GridTape. In order for the autoTEM system to control the state of the microscope without human intervention and ensure consistent data quality, we also developed customized software infrastructure piTEAM that provides a convenient GUI-based operating system for image acquisition, TEM image database, real-time image processing and quality control, and closed-loop feedback for error detection and system protection etc. During imaging, the reel-to-reel GridStage moves the tape and locates targeting aperture through its barcode. The 2D montage is then acquired through raster scanning the ROI area of tissue. Images along with metadata files are transferred to the data storage server. We perform image QC on all the data and reimage sections that fail the screening.
Volume Assembly
The images in the serial section are first corrected for lens distortion effects. A non-linear transformation of higher order is computed for each section using a set of 10 x 10 highly overlapping images collected at regular intervals during imaging. The lens distortion correction transformations should represent the dynamic distortion effects from the TEM lens system and hence require an acquisition of highly overlapping calibration montages at regular intervals. Overlapping image pairs are identified within each section and point correspondences are extracted for every pair using a feature based approach. In our stitching and alignment pipeline, we use SIFT feature descriptors to identify and extract these point correspondences. Per image transformation parameters are estimated by a regularized solver algorithm. The algorithm minimizes the sum of squared distances between the point correspondences between these tile images. Deforming the tiles within a section based on these transformations results in a seamless registration of the section. A downsampled version of these stitched sections are produced for estimating a per-section transformation that roughly aligns these sections in 3D. A process similar to 2D stitching is followed here, where the point correspondences are computed between pairs of sections that are within a desired distance in z direction. The per-section transformation is then applied to all the tile images within the section to obtain a rough aligned volume. MIPmaps are utilized throughout the stitching process for faster processing without compromise in stitching quality.
The rough aligned volume is rendered to disk for further fine alignment. The software tools used to stitch and align the dataset is available in our github repository https://github.com/AllenInstitute/render-modules. The volume assembly process is entirely based on image meta-data and transformations manipulations and is supported by the Render service (https://github.com/saalfeldlab/render).
Cracks larger than 30 um in 34 sections were corrected by manually defining transforms. The smaller and more numerous cracks and folds in the dataset were automatically identified using convolutional networks trained on manually labeled samples using 64 x 64 x 40 nm3 resolution image. The same was done to identify voxels which were considered tissue. The rough alignment was iteratively refined in a coarse-to-fine hierarchy (Wetzel et al. 2016), using an approach based on a convolutional network to estimate displacements between a pair of images (Mitchell et al. 2019). Displacement fields were estimated between pairs of neighboring sections, then combined to produce a final displacement field for each image to further transform the image stack. Alignment was first refined using 1024 x 1024 x 40 nm3 images, then 64 x 64 x 40 nm3 images.
The composite image of the partial sections was created using the tissue mask previously computed. Pixels in a partial section which were not included in the tissue mask were set to the value of the nearest pixel in a higher-indexed section that was considered tissue. This composite image was used for downstream processing, but not included with the released images.
Segmentation
Remaining misalignments were detected by cross-correlating patches of image in the same location between two sections, after transforming into the frequency domain and applying a high-pass filter. Combining with the tissue map previously computed, a mask was generated that sets the output of later processing steps to zero in locations with poor alignment. This is called the segmentation output mask.
Using the method outlined in (K. Lee et al. 2017), a convolutional network was trained to estimate inter-voxel affinities that represent the potential for neuronal boundaries between adjacent image voxels. A convolutional network was also trained to perform a semantic segmentation of the image for neurite classifications, including (1) soma+nucleus, (2) axon, (3) dendrite, (4) glia, and (5) blood vessel. Following the methods described in (Wu et al. 2021), both networks were applied to the entire dataset at 8 x 8 x 40 nm3 in overlapping chunks to produce a consistent prediction of the affinity and neurite classification maps. The segmentation output mask was applied to the predictions.
The affinity map was processed with a distributed watershed and clustering algorithm to produce an over-segmented image, where the watershed domains are agglomerated using single-linkage clustering with size thresholds (Lu, Zlateski, and Sebastian Seung 2021; Zlateski and Sebastian Seung 2015). The over-segmentation was then processed by a distributed mean affinity clustering algorithm (Lu, Zlateski, and Sebastian Seung 2021; Zlateski and Sebastian Seung 2015) to create the final segmentation. We augmented the standard mean affinity criterion with constraints based on segment sizes and neurite classification maps during the agglomeration process to prevent neuron-glia mergers as well as axon-dendrite and axon-soma mergers.
Synapse detection & assignment
A convolutional network was trained to predict whether a given voxel participated in a synaptic cleft. Inference on the entire dataset was processed using the methods described in (Wu et al. 2021) using 8 x 8 x 40 nm3 images. These synaptic cleft predictions were segmented using connected components, and components smaller than 40 voxels were removed.
A separate network was trained to perform synaptic partner assignment by predicting the voxels of the synaptic partners given the synaptic cleft as an attentional signal (N. L. Turner et al. 2020). This assignment network was run for each detected cleft, and coordinates of both the presynaptic and postsynaptic partner predictions were logged along with each cleft prediction.
Nucleus detection
A convolutional network was trained to predict whether a voxel participated in a cell nucleus. Following the methods described in (Wu et al. 2021), a nucleus prediction map was produced on the entire dataset at 64 x 64 x 40 nm3. The nucleus prediction was thresholded at 0.5, and segmented using connected components.
Proofreading
Following the methods described in Dorkenwald, et al. (Dorkenwald et al. 2020), we adapted the automatic segmentation for collaborative proofreading.
Co-registration Transform
The fiducial annotation was done using a down-sampled EM dataset with pixel sizes 256 nm (x), 256 nm (y), 940 nm (z). Though the fiducials cover the total volume of the dataset it is worth noting that below 400 µm from the surface there is much lower signal to noise in the 2P structural dataset requiring more effort to identify somata, therefore we made use of more vascular fiducials.
For calculating the transform we introduced a staged approach to separate the gross transformation between the EM volume and the two photon space from the finer non-linear deformations needed to get good residuals (a residual is the distance in microns between the location of a fiducial after co-registration and its original location; a perfect co-registration would have residuals of zero microns). This was done by taking advantage of the infrastructure created for the alignment of the EM dataset described above..
The full 3D transform is a list of eight transforms that fall into 4 groups with different purposes:
The first group is a single transform that is a 2nd-order polynomial transform between the two datasets. This first group serves to scale and rotate the optical dataset into EM space, followed by a single global nonlinear term, leaving an average residual of ∼10 µm.
The second group of transforms addresses an issue we saw in the residuals: there were systematic trends in the residual, both positive and negative, that aligned well with the EM z-axis. These trends are spaced in a way that is indicative of changing shape of the EM data on approximately the length scale between knife-cleanings or tape changes. We addressed this with a transform that binned the data into z-ranges and applied a further 2nd-order polynomial to each bin. We did this in a 2-step hierarchical fashion, first with 5 z-bins, followed by a second with 21 z-bins. These steps removed the systematic trends in the residuals vs z and the average residuals dropped to 5.6µm and 4.6µm respectively.
The third group is a set of hierarchical thin plate spline transforms. We used successively finer grids of control points of even “n x n x n” spacing in the volume. We used four steps with n = [3, 5, 10, 12]. The idea here is to account for deformations on larger length scales first, so that the highest order transforms introduce smaller changes in position. The average residuals in these steps were 3.9, 3.5, 3.1, and 2.9µm accomplished with average control point motions of 12.5, 7.5, 3.8, and 1.6µm.
The final group is a single thin plate spline transform. The control points for this transform are no longer an evenly spaced grid. Instead, each fiducial point is assigned to be a control point. This transform minimizes the residuals almost perfectly (as it should for the control points which are identical to the fiducials; 0.003µm on average, Figure 3) and accomplishes this final step by moving each data point on average another 2.9µm. This last transform is very sensitive to error in fiducial location but provides the co-registration with minimal residuals. This last transform is also more likely to create errors in regions with strong distortions, as for example the edges of the dataset.
Since the nature of transform 4 is to effectively set the residuals to zero for the control points, we used a new measure to evaluate the error of the transform. We created 2933 3D transforms, each time leaving out one fiducial and then evaluated the residual of the left-out point. We call this measure “leave-one-out” residuals and it evaluates how well the transform does with a new point.
Cell Classification
We analyzed the nucleus segmentations for features such as volume, surface area, fraction of membrane within folds, and depth in cortex. We trained an SVM machine classifier to use these features to detect which nucleus detections were likely neurons within the volume, with 96.9% precision and 99.6% recall. This model was trained based upon data from an independent dataset, and the performance numbers are based upon evaluating the concordance of the model with the manual cell type calls within the volume. This model predicted 82,247 neurons detected within the larger subvolume. For the neurons, we extracted additional features from the somatic region of the cell, including its volume, surface area, and density of synapses. Dimensionality reduction on this feature space revealed a clear separation between neurons with well segmented somatic regions (n=69,957) from those with fragmented segmentations or sizable merges with other objects (n=12,290). Combining those features with the nucleus features, we trained a multi-layer perceptron classifier to distinguish excitatory from inhibitory neurons amongst the well-segmented subset, using the 80% of the manual labelled data as a training set, and 20% as a validation set to choose hyper-parameters. After running the classifier across the entire dataset, we then tested the performance by sampling an additional 350 cells (250 excitatory and 100 inhibitory). We estimate from this test that the classifier had an overall accuracy of 97% with an estimated 96% precision and 94% recall for inhibitory calls.
ACKNOWLEDGMENTS
The authors thank David Markowitz, the IARPA MICrONS Program Manager, who coordinated this work during all three phases of the MICrONS program. We thank IARPA program managers Jacob Vogelstein and David Markowitz for co-developing the MICrONS program. We thank Jennifer Wang, IARPA SETA for her assistance.
The work was supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DoI/IBC) contract numbers D16PC00003, D16PC00004, and D16PC0005. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. HSS also acknowledges support from NIH/NINDS U19 NS104648, NIH/NEI R01 EY027036, NIH/NIMH U01 MH114824, NIH/NIMH U01 MH117072 NIH/NINDS R01 NS104926, NIH/NIMH RF1 MH117815, NIH/NIMH RF1 MH123400 and the Mathers Foundation, as well as assistance from Google, Amazon, and Intel. XP acknowledges support from NSF CAREER grant IOS-1552868. XP and AT acknowledge support from NSF NeuroNex grant 1707400. AT also acknowledges support from National Institute of Mental Health and National Institute of Neurological Disorders And Stroke under Award Number U19MH114830.
We thank Rob Young for managing the stitching and alignment pipeline at the Allen Institute for Brain Science (AIBS). We thank John Philips, Sill Coulter and the Program Management team at the AIBS for their guidance for project strategy and operations. We thank Hongkui Zeng, Ed Lein, Christof Koch and Allan Jones for their support and leadership. We thank the Manufacturing and Processing Engineering team at the AIBS for their help in implementing the EM imaging and sectioning pipeline. We thank Brian Youngstrom, Stuart Kendrick and the Allen Institute IT team for support with infrastructure, data management and data transfer. We thank the Facilities, Finance, and Legal teams at the AIBS for their support on the MICrONS contract. We thank Stephan Saalfeld, Khaled Khairy and Eric Trautman for help with the parameters for 2D stitching and rough alignment of the dataset.
We thank Zane Hanson and Justin Singh for their contribution to manual matching of functional units to EM nuclei, and Donnie Kim for his contribution to pupil tracking. We also thank Rajkumar Raju for his contribution to parametric stimuli development. We thank Aaron Mok and Dimitri Ouzounov for their contribution to three-photon imaging development.
We thank Garrett McGrath for computer system administration, and May Husseini and Larry and Janet Jackel for project administration at Princeton University.
We thank Brock Wester, William Gray-Roncal, Sandy Hider, Tim Gion, Daniel Xenes, Jordan Matelsky, Caitlyn Bishop, Derek Pryor, Dean Kleissas, Luis Rodriguez and Miller Wilt from John Hopkins University Applied Physics lab for providing data assessments on the neural circuit reconstruction and infrastructure through BOSSdb
We thank Frances Chance, Brad Aimone, and everyone at Sandia National Laboratories for their support and assistance.
We would like to thank the “Connectomics at Google” team for developing Neuroglancer and computational resource donations, in particular J. Maitin-Shepard for authoring neuroglancer and help creating the reformatted sharded multi-resolution meshes and imagery files used to display the data. We would like to thank Amazon and the AWS Open Science platform for providing computational resources. We’d like to also thank Intel for their assistance.
We thank the Allen Institute for Brain Science founder, Paul G. Allen, for his vision, encouragement and support. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/IBC, or the U.S. Government.
Footnotes
The manuscript has been corrected to include Marc Takeno as an author and update the acknowledgements of external parties.