Abstract
We have achieved a three fold increase in the speed of transmission electron microscopy by using a beam deflecting mechanism to enable highly efficient acquisition of multiple image tiles for each motion of the mechanical stage. For millimeter-scale areas, the duty cycle of imaging doubles and exceeds 30%, yielding a net average imaging rate of 0.3 gigapixels per second.
Main text
Volume electron microscopy (EM) is currently the only approach that has been used to reconstruct a connectome – a complete map of neural connectivity at synaptic resolution. Serial section transmission electron microscopy (ssTEM) is a classic volume EM technique used for 3D reconstruction of neural tissues at small scale1–3. In the 1980s, ssTEM was used for mapping the first whole-brain connectome of C. elegans4. Later on, ssTEM was used to acquire 3D EM datasets of multiple C. elegans brains5,6, an entire fly larval brain7, normal and pathological retinas8,9, and mouse cortex10,11. Recently, automation and parallelization of TEM10,12,13 have allowed imaging of ever larger volumes, such as a complete adult flybrain14 and a cubic millimeter volume of mammalian cortex15.
The previous state of the art in high throughput TEM was a system at the Allen Institute for Brain Science12. This achieves high imaging speed through several innovations. A reel-to-reel tape translation system allows automated delivery of sections based on GridTape technology13. A 50 MPixel CMOS camera with a low distortion lens achieves a burst imaging rate of 0.5 gigapixels per sec (GPix/s, imaging only). A scalable software infrastructure allows closed-loop workflow management based on real-time image processing16. However, imaging time for a cubic millimeter volume is still 6 months with multiple TEMs12 and there is a growing need to image much larger datasets17.
Imaging a square millimeter area requires thousands of x-y stage translations to acquire image tiles that are later stitched together to form a multi-tile image. Because the time to image a tile has become so fast the stage translations require more time than the image capture itself12. A massive number of positional translations – typically ~10,000 per mm2 – also rapidly consumes piezo stage life of ~10 billion cycles of the crystal per year when operated 24/7. Therefore, to advance the state of the art, it is essential to reduce the overhead due to stage translations.
Here we report the development of a beam deflection mechanism (Cricket™) to significantly increase TEM imaging throughput (bdTEM, beam deflection). In addition, the microscopes are integrated with an automated reel-to-reel tape translation system12,13 and a 50 MPixel CMOS camera12. The bdTEM achieves the fastest TEM imaging rate to date, a three fold increase over the previous TEM with the same camera12.
Beam deflection (Cricket)
We repurposed electromagnetic lens deflectors in the TEMs, above the object plane and below (Fig. 1a). The deflectors above the object plane shift the electron beam over the specimen in a matrix of 3 × 3 tiles and simultaneously the deflectors below the object plane precisely de-scans the image back onto the TEM camera. The beam deflection allows acquisition of 9 image tiles without moving the stage, eliminating 8 out of 9 stage movements (Fig. 1b-c, Extended Data Fig. 1b). Overhead for stage motion is therefore significantly reduced, accounting for only 7% of montage imaging time with an additional 9% of Cricket settling time (Fig. 1d). As a result, the imaging duty cycle increases from a previous peak performance of 15%12 to over 30% of total time per section in our system (Methods).
a, The TEM is a refurbished JEOL 1200EX-II with 120 KV accelerating voltage. The custom modifications include a beam deflection mechanism (Cricket, Voxa), an advanced reel-to-reel tape translation system with a dual-axis piezo-driven fast stage (GridStage, Voxa), and a high speed CMOS camera (CB500MG-CM, XIMEA) with improved lens design (NanoSprint50M-AV, AMT). b, Scanning sequence of tiles in a supertile. c-d, Distributions of stage step-and-settle time in × (c) and y (d) axes.
a-b, Cricket Schematic. At a stage position, Cricket sequentially scans each tile of a 3 × 3 matrix pattern on the sample. The tiles can be stitched together in real-time, producing a “supertile”. Each square represents a single tile and 9 tiles compose a supertile with overlap between tiles. Neighboring supertiles (different colors) can be stitched together to form a montage image. c, Overview of a section on a 2 mm × 1.5 mm slot imaged using Cricket at low magnification. d, Imaging speed based on acquisitions at 3 nm/pix (Methods). The bar lengths are proportional to the contribution of each overhead to imaging speed. Computational overhead of acquisition (“acq software”) includes communications between different components, software time jitters, computing image statistics, on-the-fly stitching of subtiles to supertiles, and image storage. e, A fully stitched montage of a hippocampus section (1 mm2) imaged with a Cricket-equipped TEM. The montage includes 4320 tiles (or 480 supertiles). f, A montage of 6 stitched supertiles (from red outline in e). g, A Cricket supertile consisting of 9 tiles (from orange outline in f). Each tile has 6,000 × 6,000 pixels and the overlap between neighboring subtiles of a supertile is 15% (~900 pixels). h, A single image tile (from green outline in g). i, High-magnification EM images with 120 ms (upper, from blue outline in h) and 40 ms exposure time (lower).
Optics
The detector optics of the TEM is upgraded with a compact light-optical lens assembly and a high speed CMOS camera that yields a frame size of 6,000 × 6,000 pixels (or 36 MPix). The sensor sits near the film plane of a conventional TEM (Extended Data Fig. 1a), eliminating the impractical need of an extended vacuum column required by the past generations of large pixel-count detectors10,12. Images acquired at an exposure time of 40 ms have sufficient resolution for synapse identification and reconstruction of neuronal morphology (Fig. 1e-i, Supplementary video 3). As a result, the burst imaging rate (0.9 GPix/s, imaging only) of bdTEMs is 1.8 times that of the previous fastest imaging rate12 (0.5 GPix/s).
Section transition and stage
The TEMs are integrated with a previously reported reel-to-reel tape translation system and a two-axis piezo-driven stage12,13. This reel-to-reel tape translation system allows automated section exchange and barcode-based slot identification using GridTape technology12,13. In previous volume TEM techniques, stage step-and-settle time ranges from under 50 ms to 100 ms for one axis12,14. The stage in our TEMs achieves a step-and-settle time of ~57 ms on average on both axes (Extended Data Fig. 1c-d), increasing efficiency on the remaining stage motions.
These modifications dramatically increase imaging throughput. Acquisition of a 1 mm2 section takes 8.6 minutes including imaging, stage motion, and section transition (net average imaging rate of 0.3 GPix/s at 3 nm/pix, Fig. 1d, Table 1, and Supplementary Video 1). The net imaging rate is 3 times the peak imaging rate of the previous fastest TEM at 0.1 GPix/s with the same 50 MPix camera12. The previous imaging rate12 includes overhead for image quality control and post-processing. These functionalities have not yet been integrated into our acquisition software but in principle could be performed during the overhead for acquisition software and section transition.
We set out to test whether the TEMs can be used to acquire a 3D EM volume. The reel-to-reel sample delivery system requires serial ultrathin sections to be cut and collected on GridTape12,13. Based on previous designs12,13, we modified an ATUMtome (RMC/Boeckeler) to be compatible with an ultramicrotome (Leica UC7) and GridTape, and used the automated sectioning system to collect serial ultrathin sections of mouse hippocampus tissue. Imaging a series of sections involves cycling through the following steps: (1) translate GridTape to the targeted slot guided by index barcodes; (2) acquire an overview image of the section at low magnification (Fig. 1c); (3) extract ROI from the overview image and set imaging parameters (e.g. illumination correction and autofocus); (4) montage imaging of the ROI at high resolution. Basic software has been implemented to run the workflow (Supplementary video 1). With 10% overlap between supertiles and 15% overlap between subtitles within a supertile, all tiles of a common section of 1 mm2 in size can be stitched together (Fig. 1e-i). Consecutive sections are aligned to assemble a volume (Supplementary Video 2-3). Ultrastructural features such as postsynaptic densities and vesicles are clearly defined (Fig. 1i). Small-diameter axons and dendrites can be followed across sections. These data demonstrate that sequential sections imaged with the bdTEMs can be assembled into a 3D volume for connectomic application.
An array of four microscopes have been fully installed in our facility (Extended Data Fig. 2), some of which have been reliably acquiring data for many months with reasonable microscope downtime12. By estimate, if the four bdTEMs are running 24/7, imaging a cubic millimeter will take about a month (Table 1). However, non-stop long-term imaging requires further development of automation software that can handle image quality control, workflow management, image and section databases, and stitching and alignment. A scalable architecture with these functionalities has been developed16 but needs to be adapted to integrate with bdTEMs. On raw imaging speed, breakdown of various overhead sources (Fig. 1d) points to several avenues of future improvements. Upgrade of the optics with a 100 MPix camera will in principle double the burst imaging rate. Additional increase of imaging speed can be gained by refactoring frequently used tasks in the acquisition software to low-level programs, reducing tile overlap, enlarging pixel size, and increasing efficiency on section transition.
a, Dimensions of a microscope room that houses an array of 4 bdTEMs with an inside room for sectioning.
b, A photograph of 4 bdTEMs that are installed in the same microscope room (a).
Volume EM is dichotomized into SEM-based and TEM-based methods, and both have been shown to be capable of acquiring large-scale datasets (MultiSEM18 and GridTape-based TEM15). Here we have further improved the TEM imaging speed to 0.3 GPix/s. Similar to the previous TEMs12,13, our modifications are based on a retrofitted JEOL 1200EX-II microscope and cost for the entire system (< $500,000) is significantly lower compared to equivalent imaging systems, permitting acquisition of multiple TEMs for an imaging facility (six at the Allen Institute for Brain Science12 and four at Princeton University). The current system, however, requires specialized films on GridTape, which scales linearly in cost with the number of sections (USD $4 - $8 per slot); but improving film-making methods and economies of scale may decrease the cost in the future. Crucially, both imaging methods currently depend on cutting and collecting a large number of ultrathin sections. Sectioning-related artifacts (folds, cracks, knife marks, etc.) are very common, though automated reconstruction methods that are robust to some defects have been developed19,20. Wide-area milling of imaged tissue was recently developed to circumvent ultrathin sectioning21 and, if integrated with a MultiSEM, can potentially scale up connectomic volumes beyond petascale. In addition, the reliability of serial sectioning can be improved by magnetic section collection22. For TEM, one proposal to improve reliability of sectioning is to increase the thickness of serial sections, which are then imaged with tomography to recover a finer axial resolution23. Whether SEM or TEM-based techniques is the method of choice for exascale datasets remains to be determined.
Methods
Sample preparation and sectioning
All procedures were carried out in accordance with the Institutional Animal Care and Use Committee at Princeton University. Mice (C57BL/6J-Tg(Thy1-GCaMP6f) GP5.3Dkim/J, Jackson Laboratories, 028280) aged 6 months were transcardially perfused with a fixative mixture of 2.5% paraformaldehyde and 1.3% glutaraldehyde in 0.1M Cacodylate with 2mM CaCl2 pH 7.4. The brain was extracted and post-fixed for 36 hrs at 4°C in the same fixative solution. The perfused brain was subsequently rinsed in 0.1M Cacodylate with 2mM CaCl2 for 1 hr (3 × 20 mins) and 300 μm coronal sections were cut on a Leica Vibratome. Sample blocks were cut out and stained based on a modified reduced osmium treatment (rOTO) protocol with the addition of formamide15,24. The tissue blocks were first en bloc stained with 8% formamide, 2% osmium tetroxide, 1.5% potassium ferrocyanide for 3 hours. Subsequently, the samples were immersed in 1% TCH (94 mM) 50 °C for 50 mins, followed by a second step of 2% osmium staining for 3 hours. The sample was placed in 1% uranyl acetate overnight at 4°C, followed by lead aspartate (Walton’s, 20 mM lead nitrate in 30 mM aspartate buffer, pH 5.5) at 50 °C for 2 hours. After washed with water (3 × 10 mins), samples proceeded through a graded acetonitrile dehydration series (50%, 75%, 90% w/v in acetonitrile, 10 minutes each at 4 °C, then 4 × 10 minutes of 100% acetonitrile at room temperature). After a progressive resin infiltration series (33% resin:acetonitrile, 50% resin:acetonitrile, 66% resin:acetonitrile, 8 hours each), the sample was incubated in fresh 100% resin overnight and the resin was cured in the oven at 60 °C for at least 48 hrs.
GridTape (Luxel Corporation) contains regularly spaced apertures with plastic film substrates for serial sections12,13. The films are transparent to electrons, and therefore are compatible with TEM imaging. We combined an automated tape collecting system (ATUMtome, RMC/Boeckeler) and an ultramicrotome (UC7, Leica) to create an automated tape-based sectioning and collection system (Automated Tape Collecting Ultramicrotome, ATUM) for GridTape, based on previous designs12,13. Our custom ultramicrotome setup also includes a computer-controlled, high-precision motorized stage, monitoring of temperature and humidity, and three cameras to monitor the collection process. Additionally, we have built closed-loop control software to phase-lock aperture movement with cutting, in order to center sections collected in the apertures on GridTape. After resin embedding, the ultrathin sections were then cut at a nominal thickness of 40 nm and automatically collected onto a GridTape.
Reel-to-reel translation system and GridStage
The reel-to-reel translation system and GridStage (Voxa) have been described previously12. Briefly, Gridstage Reel comprises three major components: a stage cartridge, a reel storage and delivery system, and an airlock assembly enabling cartridge loading. A 3D rendering of the GridStage Reel system is shown in Extended Data Fig. 1, attached to a JEOL 1200EX-II microscope chassis. GridStage’s two-axis stage cartridge is set to a fixed height to enable the imaging environment to stay the same from section to section as they are delivered into the imaging area by conveyor. Each axis is driven by a precision closed-loop piezo motor. The precision is ~50 nm, and absolute positional accuracy on the sample is typically between 100 - 200 nm. This mechanism positions the sample accurately and reliably within the requirements of montage image overlap reproducibility, even given electron-optical environment variability (e.g. charging and sample morphology changes) of different samples in the volume. The cartridge incorporates an accurate barcode reading and clamping mechanism to precisely position and identify slots on GridTape, with read accuracy greater than 99.9% for fresh GridTape. This enables precise tracking of the current tissue section under first-time montage imaging or during re-imaging operations. The precision of the GridStage sample ID subsystem supports either sequential or fully random-access sample delivery modes.
The stage has fast responsiveness, with roundtrip communication to the drive system on the order of a few milliseconds, and typical step-and-settle times (>95% of moves) occurring in under 60 ms (Extended Data Fig. 1c-d). The average step-and-settle time is similar to and slightly improved over a previously reported fast Piezo-driven stage14. Voxa reels are housed in robotic delivery and take-up conveyor systems connected to the microscope’s vacuum, and can accept a GridTape reel with up to 5,500 sections spaced 6 mm apart. A continuously-monitored belt tension meter ensures the GridTape translates within specified operating parameters, and supports safe transit of the samples into and out of the imaging area. The GridStage reel sample delivery system can deliver sequential sections as fast as once every two seconds, for quick survey modes.
Lens assembly and camera
The electron optics of each of our TEM systems consists of a custom phosphor scintillator of 75 mm in diameter, a lens assembly (AMT, NanoSprint50M-AV), and a high speed CMOS camera (XIMEA, CB500MG-CM). The pickup area of 47 × 47 mm on the phosphor screen is near the film plane of the JEOL 1200EX-II TEM and therefore minimizes image distortion at the bottom of the column and matches the nominal magnification setting of the microscope platform. Conveniently, this setup eliminates the need for a lengthened column that previously requires building scaffolds and is impractical for many facilities10,12,14. In addition, compared to the previous design of an extended column, the smaller scintillator and compact lens in our TEMs allows good signal-to-noise with lower dose or with shorter exposure times leading to more efficient use of available beam electrons. The CMOS camera has a PCIe interface that supports fast data transfer rates. Overall, the lens and camera supports nominal image acquisition with a frame size of 6,000 × 6,000 pixels at a shortest exposure time of 40 ms. In the future, an upgrade of a 100+ Mpix CMOS camera with backside illumination could potentially lead to another couple-fold increase of imaging rate12.
TEM imaging
Some components of the TEM systems have been briefly presented before25. The Beam deflection mechanism (Cricket) for TEM has been prototyped previously12 but not demonstrated for large-scale imaging. For imaging, a reel containing samples on a GridTape are first loaded into the reel housing of the reel-to-reel system (Extended Data Fig. 1), which is connected with the TEM column vacuum. After loading, the microscope is pumped down to reach the vacuum level of ~1E-7 Torr. Lanthanum hexaboride crystal (LaB6) filaments were used due to their high electron flux per unit area and a longer lifetime (1,000 hrs or more as opposed to 200 hrs for a tungsten source) reducing downtime needed for filament change. Crucially LaB6 filaments depend more sensitively on a good vacuum pressure for good stability and lifetime (low 10-7 Torr). It typically takes about half a day when loading a new sample reel to pump down the column to the base operating pressure for LaB6. We then increase the high-tension voltage to 120 KV, turn on filament current and then perform alignments on various components of the TEM, following routine technical instructions for TEM operations. These procedures take half a day and only need to be performed after installation of a new filament or loading of a new tape. Calibration of reel-to-reel systems involves tape and tension calibration for reliable translation and barcode reading, which typically takes ~5 minutes and is only done when first loading a reel. Cricket alignment is performed to ensure each sub-tile has sufficient overlap (~15%) with neighboring sub-tiles and that there is minimal distortion across the field between images from different sub-tiles to facilitate efficient stitching and reconstruction. Cricket alignment is usually stable over several months.
The TEMs are compatible with GridTape13, which has regularly spaced 2 × 1.5 mm2 apertures in aluminum-coated polyimide tape. Each aperture can be identified by a unique barcode milled on the tape. The tape is coated with a 50 nm-thick film (Luxel Corporation) that spans the apertures and serves as support for sections. The steps to perform before imaging every section include: deliver a new aperture into the field of view using the reel-to-reel system, locate the aperture center using stage algorithmically, find the illumination center of the electron beam (which is often off-center due to magnetic hysteresis from magnification change or change in charge equilibrium state of the new aperture), adjust camera gain control, extract ROI, and autofocus at high magnification. These are done for every aperture with a tissue section, and typically take 2 - 3 minutes per aperture (Table 1). Software (Blade, Voxa Inc.) has been developed that can control the Reel-to-reel system, GridStage, and TEM automatically. The software has lower-level functions that execute each step in the workflow (e.g. travel to a specific aperture in the GridTape reel, move the stage, adjust focus of the TEM). The low-level software supports end-user scripting and control of application-specific microscope workflows via the Blade API supplied to the user. Additionally, the Blade software allows initialization of imaging remote control over IP of all operations and multiple user-defined imaging modes - e.g. continuous batch imaging, re-imaging, and selective or random sampling.
The acquisition computer for each microscope has an intermediate storage of 16 - 32 TB SSDs via a PCIe card with 4 USB3.0 ports, for ~24 hours of imaged data. Each USB port can support up to 10 Gb/s of transfer speed. After caching on the temporary storage on SSD, the data are then transferred via a 10 Gb/s network connection to an on-premise petascale cloud storage in a separate building. Data transfer is done using CloudFiles (https://github.com/seung-lab/cloud-files).
In general, imaging rates (Fig. 1d, Table 1) are computed as the number of pixels that are acquired in a given amount of time to image a section. Burst imaging rates are calculated from the number of pixels in a frame over camera exposure time per frame (6,000 × 6,000 pixels in 40 ms). Montage imaging rates are calculated from the total number of pixels of all images (i.e. number of pixels in a tile multiplied by number of tiles) over time, from the first image to the last image of a montage acquisition. Sources of overhead for montage imaging rates include imaging, stage motion, beam deflection, computational overhead of the acquisition software, and computing basic image statistics (Fig. 1d). Net imaging rates are computed from the same number of pixels as in montage imaging rate, including all overhead from montage imaging, but additionally includes overhead for section transition (e.g. low-magnification overview of sections on GridTape, ROI extraction, autofocus, automatic beam centering). Effective imaging rates are calculated from the total number of pixels of a montage acquisition with overlap pixels only counted once, over the same overhead as in the net imaging rate. Time for a montage acquisition varies from section to section by ~30 MPix/s (s.d.) depending on variable computational overhead in the acquisition software. Peak imaging rate is taken from an acquisition with the shortest imaging time. Average imaging rates are taken from a typical production acquisition (an average or 10 or more acquisitions). Average transition time is the average taken from 10 consecutive automatic acquisitions.
Imaging duty cycle is computed as the fraction of time for image acquisition (number of tiles × exposure time per tile) out of total time per section. For example, at 3 nm/Pixel the image acquisition time is 2.9 minutes (4,356 tiles at 40 ms per tile), which is 33.7% out of total time per section (8.6 minutes). Similarly, the imaging duty cycle at 4 nm/Pixel is 31%. In comparison, the peak performance of the previous state of the art system12 has an imaging duty cycle of 15% (2,600 tiles at 50 ms per tile out of 14 minutes of total time).
Stitching and alignment
Image stitching was performed with the AlignTK software10 (https://mmbios.pitt.edu/aligntk-home). Image alignment was performed with a custom software pipeline20.
Supplementary video 1. Screen recording of an acquisition at a peak imaging rate.
Supplementary video 2. Aligned images across 55 sections acquired at 120 ms exposure time.
Supplementary video 3. Aligned images across 20 sections acquired at 40 ms exposure time. https://doi.org/10.6084/m9.figshare.21614058
Author contributions
C.S.O. and A.A.W. conceived the study. Z.Z. developed the conceptual framework for quantifying the performance. D.W.T. and H.S.S. acquired funding and supervised. A.A.W. planned the construction of the facility and managed the installation and testing of the first instrument. A.A.W. registered and stitched the initial datasets. Z.Z., C.S.O., and R.A.K. developed software for TEM imaging. Z.Z., A.A.W., and E.W.H. developed software and hardware for serial sectioning. W.M.S. developed software for data transfer. Z.Z. performed image stitching. Z.Z. and N.K. performed image alignment. Z.Z. acquired and analyzed data. Z.Z. and C.S.O. carried out validation and evaluation. Z.Z. and H.S.S. drafted the manuscript with input from all authors.
Competing interest
N.K. and H.S.S. disclose financial interests in Zetta AI LLC. C.S.O. and R.A.K. disclose financial interests in Voxa. A.A.W. is a founder and owner of ariadne.ai ag (Switzerland).
Acknowledgments
The Cricket system was initially prototyped and tested by Voxa with the Allen Institute for Brain Science12. We thank R. Clay Reid, Daniel J. Bumbarger, Wenjing Yin, Derrick Brittain, Marc Takeno, Nuno Macarico da Costa, David Hildebrand for advice on microscopy and serial sectioning setup; Stephan Y. Thiberge for help in the planning and construction of the TEM facility; Lawrence Own and Teddy DeRego for their development and support in GridStage and Cricket hardware and software; Wei-Chung Allen Lee and Jasper Phelps for advice on image stitching; John Price for modification of ATUMtome. This work was supported by the NIH grants 1S10OD023602-01A1 and 1U19NS104648 and the Simons Collaboration on the Global Brain. The authors acknowledge the use of Princeton’s Imaging and Analysis Center, which is partially supported through the Princeton Center for Complex Materials (PCCM), a National Science Foundation (NSF)-MRSEC program (DMR-2011750).
References
