Abstract
Multi-channel detection in single-molecule localization microscopy (SMLM) greatly increases information content for various biological applications. Here, we present globLoc, a graphics processing unit (GPU) based global fitting algorithm with flexible PSF modeling and parameter sharing, to extract maximum information from multi-channel single molecule data. We show, both in simulations and experiments, that global fitting can substantially improve the 3D localization precision for biplane and 4Pi SMLM and color assignment for ratiometric multicolor imaging.
Single-molecule localization microscopy (SMLM) achieves nanometer superresolution and has become an important method for structural cell biology. Various extensions of SMLM using two or more detection channels are instrumental for this success, as they greatly increase information content that can be extracted from samples: Multi-color SMLM imaging of proteins labeled with fluorophores of different color can probe their spatial relations and interactions. It is usually realized using two spectral channels1–3 or one spatial channel combined with spectral detection in a second channel4. Three-dimensional (3D) SMLM techniques using two or more detection channels, such as biplane5 or multi-plane6 detection, self-bending point spread functions7 (PSFs), supercritical-angle fluorescence detection8,9 and multi-phase interference10,11, are powerful in investigating the intrinsic 3D organization of biological structures. Two or more fluorescence polarization channels are used to probe the orientation of fluorophores12, offering insight into the orientation of proteins in a molecular machinery. Recently, modulation enhanced localization microscopy that uses patterned excitation with rapid detection of different phases of the pattern on multiple parts of a camera, was used to increase the resolution of SMLM by a factor of two13–16.
Compared to the single-channel SMLM, data analysis for all these methods is complicated by the fact that measures from two or more channels have to be combined to result in the additional information (color, z-position, polarization state, interference phase, etc.). Typically, this is achieved by first fitting the fluorophores individually in each channel to extract corresponding parameters, and then combining the returned parameters from different channels to obtain the extra information1-16. Separate fitting of an individual fluorophore present in two channels is not optimal, as we neglect the information that the fitting parameters (e.g., 3D positions and photons) are highly correlated. If instead we were to use a global fitter that links the correlated parameters across different channels, this would decrease the number of fitting parameters, improve precision and robustness of the fit and avoid ambiguity when pairing corresponding parameters. Additionally, it would allow precise analysis of a fluorophore that is very dim in one of the channels and thus would escape molecule detection when fitted separately. In spite of the many benefits of analyzing separate channels simultaneously, global fitting is not widely used for the multi-channel single molecule localization. First approaches for global fitting17–20 lack flexibility with respect to the PSF models and fitting parameters. They are often designed for a specific imaging modality and difficult to be integrated into complete analysis workflows to be of general use.
Here, we developed globLoc, a general data analysis workflow and easy to use software for global fitting of single molecule data detected in separate channels. Its optimized analysis pipeline includes: generation of a precise transformation among the channels, calibration of a global multi-channel PSF, a GPU based global fitter that achieves maximum accuracy (Supplementary Fig. 1, 2 and 3) and ultra-fast fitting speed (Supplementary Fig. 4), as well as post-processing routines to extract the additional information (z, color, interference phase, polarization, etc.). Both, in simulations and on experimental data, we showed that global fitting indeed leads to a substantially improved localization precision for biplane and 4Pi-SMLM and color assignment in multi-color astigmatic SMLM.
We now give an overview of the globLoc analysis workflow (Figure 1a, Supplementary Fig. 5), and the details can be found in the Methods. We describe it using as an example dual-channel single molecule data. The extension to multi-channel data is straightforward. We first generate a global multi-channel experimental PSF model from image stacks of beads immobilized on a coverslip. To this end, we first calculate spline PSF models for each channel independently21 and fit each channel individually with the corresponding PSF model to obtain the precise bead positions. From corresponding bead positions in the two channels, we calculate the transformation between the channels. We then use cubic spline interpolation to register and average many bead stacks21, while keeping the fixed spatial relationship between the channels described by the transformation. Optionally, we re-calculate the transformation based on the actual SMLM experiment to account for channel drift. For this, we fit a sub-set of single molecule data in each channel separately using the corresponding PSF model and calculate the transformation based on the fitted coordinates. Besides using an experimental PSF model, our software also supports global fitting with a Gaussian PSF model.
After calibration of the multi-channel PSF model and the transformation between different channels, the next step of the workflow is to perform global fitting to jointly analyze the multi-channel data using maximum likelihood estimation (MLE). On a standard GPU (NVIDIA RTX3090), our implementation reached ∼ 35,000 fits/s for regions of interest (ROI) with a size of 13×13 pixels, while the speed was ∼ 1,000 fits/s on a CPU (Intel Core i7-8700, Supplementary Fig. 4). On simulated data for biplane SMLM, global fitting reached the Cramer-Rao-Lower-Bound (CRLB) in 3D over a large axial range (±600 nm, Supplementary Fig. 1a).
As globLoc is very flexible to link or unlink parameters between different channels, we compared the localization precision in the conditions of individual fit, global fit with only linking xyz positions and linking xyz positions plus photons per localization. Compared to individual fitting of the channels followed by CRLB-weighted averaging of positions9 (Supplementary Note 1 and 2), globLoc achieved about 1.5 times better z localization precision (Figure 1b and Supplementary Fig. 1b) and more robust parameter estimation (Supplementary Fig. 1c). This resolution improvement was further confirmed by participating in the continuously running 2016 SMLM Software Challenge22, in which globLoc improved the 3D localization precision by almost a factor of two on biplane data, compared to the second-best performing algorithm LEAP (Figure 1d). Our own comparison on the training data set (simulated microtubules) showed a clear improvement compared to the popular SMLM analysis software ThunderSTORM23 (Figure 1e). The improvement of globLoc compared to ThunderSTORM was even more apparent for when we analyzed experimental SMLM data of nuclear pore complex (NPC) protein Nup96, which we used as a reference standard24. In contrast to ThunderSTORM, globLoc was able to clearly resolve the two-ring structure of the NPC (Figure 2a). This is likely not only due to a better localization precision, but also an improved robustness of the fit.
GlobLoc is not limited to 2 channels. We implemented four-channel fitting for 4Pi-SMLM with multi-phase interference, using an advanced experimental 4Pi-PSF model that we developed recently25 (Supplementary Note 3). By fitting all four phase images globally with such a spline-interpolated experimental PSF model, globLoc achieved the CRLB in all dimensions, and greatly improved precision as well as accuracy compared to the state-of-the-art analysis (Figure 2b and Supplementary Fig. 3). As for biplane data, we found that additionally linking the photon number between different channels improved the localization precision in z by up to 1.5 times compared to only linking xyz (Supplementary Fig. 3).
Ratiometric multicolor SMLM images two or more dyes with overlapping emission spectra in two spectral channels and assigns the color of single molecules based on the relative number of photons detected in each channel (Supplementary Fig. 6a and b). It has many advantages over conventional multicolor super-resolution imaging using dyes with well separated emission spectra1,3,26: 1) it has a negligible channel shift and chromatic aberration; 2) many of the best “blinking” dyes have similar emission spectra in the dark red range and are compatible with similar imaging conditions; 3) multi-color imaging can be performed simultaneously with one excitation laser. A key challenge for ratiometric color assignment is to precisely determine the photon number of the single molecules to distinguish their color. By using salvaged fluorescence reflected by the main excitation dichroic mirror, Zhang et al. have shown 3 color superresolution imaging of biological structures in 3D at 5-10 nm localization precision using 4Pi-SMS microscopy3. However, the salvaged fluorescence was only used for color assignment and did not contribute to the molecule localization.
Global fitting with globLoc improves the accuracy of determining the photons per localization (Supplementary Fig 2b) and thus the color assignment in both simulation and experiment (Supplementary Fig. 6, 7 and 8), while utilizing all detected photons for localization. To exploit our finding that linking photon numbers across channels increases the accuracy, we implemented a fitting approach in which globLoc fixes the relative photon numbers across the channels to different pre-calculated values and chooses the solution with the maximum likelihood (Supplementary Fig. 7). This approach reduced crosstalk during color assignment and minimized rejection of single molecules with close intensity ratios. It also makes a post-processing step for color assignment obsolete (Supplementary Fig. 8).
These innovations of GlobLoc enabled us to image and faithfully distinguish a record of 4 colors simultaneously in ratiometric 3D SMLM (Figure 2c, Supplementary Fig. 9 and Supplementary Video 1) and image Nup96, Nup62, Elys and WGA within single NPCs labeled with the dyes AF647, DY634, CF660C and CF680 with no apparent cross-talk. We averaged 200 NPC images by registering the Nup96 structures that we used as a reference27. This protein density map shows the average positions of the four NPC proteins, with Nup96 forming two rings with an 8-fold symmetry, Elys forming a large ring and Nup62 and WGA localizing at the central channel of the pore.
To summarize, we demonstrated that linking shared parameters during multi-channel single molecule localization substantially improves localization accuracy and reduces color assignment crosstalk. GlobLoc is fully integrated in SMAP28, allowing anyone to directly and easily use its full functionality (complete multi-channel calibration pipeline, versatile PSF model, flexible parameter sharing and fast fitting speed accelerated by GPU). In addition, as it is published as open-source with example code, it can also be easily used for custom software. We believe that globLoc will enable many groups to substantially improve their analysis workflows for multi-channel SMLM.
METHODS
Calculation of multi-channel transformation
Global fitting of multi-channel data relies on knowing the precise transformation among the channels. We developed a routine to calculate transformations from coordinates (bead positions or positions of single fluorophores) that we used during the generation of the multi-channel PSF model and for global fitting of multi-channel single molecule data. We describe our algorithm for a two-channel transformation (reference and target channel ). A multi-channel transformation is represented as several two-channel transformations from all target channels to the same reference channel. Our algorithm is as follows (Figure 1a and Supplementary Fig. 5):
Obtain approximate transformation T0. This can be the transformation calculated in a previous experiment. Alternatively, we calculate it by first binning the coordinates in super pixels with a size of 50 nm. Then, we calculate the image cross-correlation and determine the position of the cross-correlation peak with sub-pixel accuracy from the position of the brightest pixel in a 4-fold upscaled image calculated by Gaussian filtering followed by cubic interpolation.
We transform all target coordinates to the reference channel using .
We link coordinates in the reference and target channel if they are closer than a maximum distance . For fluorophores from SMLM experiments we only link coordinates from the same frame.
We calculate the precise transformation T based on the linked and as anchor points. Usually, we use a projective transformation where T is represented by a 3 × 3 matrix, but we can use all transformations supported by Matlab, e.g., polynomial transformations.
If necessary, we repeat steps 3 and 4 with reduced ρ.
Generation of multi-channel PSF models
Our algorithm to generate multi-channel PSF models from bead stacks is an extension of our work on generating single-channel experimental PSF models21. Again, we illustrate the steps of our algorithm on the dual-channel example, an extension to N channels is straight forward (Figure 1a and Supplementary Fig. 5):
We find candidate bead positions in each channel by calculating the mean image over all z-positions, Gaussian filtering and finding of local maxima above a user-defined threshold. These candidate positions are integers in the unit of camera pixels.
If no transformation among the channels exists, we first generate single-channel PSF models for each channel separately. We then fit the bead images using these new PSF models and finally use the fitted localizations to calculate T as described above.
We transform the coordinates of the candidate bead positions from the reference to the target channel: . These target coordinates are continuous coordinates; thus we calculate the nearest integer pixel position by rounding the transformed coordinates and calculating the shift between the rounded and original transformed coordinates .
We cut out ROIs around and out of the bead stacks and shift the ROIs of the target channel by using cubic interpolation. If the target channel is mirrored with respect to the reference channel, we mirror the target ROI. This ensures that beads in both channels are shifted in the same direction during registration. Finally, we concatenate image stacks in both channels to form a single 3D array.
We create an initial template by averaging the 3D arrays over all beads and use 3D cross-correlation to register all beads to this template.
We reject those beads that have an insufficient overlap with the template (quality control) and calculate the next template as the average of the remaining shifted beads. We then register the central part of each bead to the new template.
We normalize the beads by the sum of the central slice of the reference stack.
We slightly filter the PSF models in z with a smoothing bspline and calculate a cspline representation for each channel.
To validate the PSF calibration, we fit each bead in the bead stack and compare the fitted z position with the true z position as denoted by the frame in the image stack.
Extraction of multi-channel single molecule data
We implemented the workflow for global fitting of single molecule blinking events in the following way (again illustrated for two channels):
We calculate the global PSF model as described above from bead stacks.
Optionally, especially if we did not acquire bead stacks on the same day as the SMLM measurements, we calculate an improved transformation by fitting single molecule localizations in each channel independently and then using these localizations as anchor points to calculate T as described above. Otherwise, we use the transformation from the bead calibration.
We find candidate peaks in all channels using a difference of Gaussian filter and maximum finding. We then transform all candidates back to the first channel and average close-by candidate positions to obtain the coordinates of the candidates in the reference channel. Finally, we round to the nearest integer pixels to obtain .
As described for the beads, we transform the candidate positions to the target channels and calculate the shift between the rounded and original transformed coordinates .
Then we cut out ROIs around and . If the two channels are mirrored, we additionally mirror the ROIs and .
Maximum likelihood estimation of multi-channel single molecule data
We use a maximum likelihood estimator that jointly optimizes the combined likelihood across different channels. The objective function for MLE across different channels is given by: Here, Mki is the measured photon number in the kth pixel of the ith channel. μki is the expected photon number in the kth pixel of the ith channel. Similar to previous implementations21,29,30, we used a modified Levenberg-Marquardt (L-M) algorithm to minimize (Supplementary Note 4). For the multichannel nonlinear optimization process, the parameters can be classified as either shared (global) or non-shared (local) parameters, θ ∈ (θp, θqi). Here, θp is the set of global parameters and θqi is the set of local parameters of ith channel. The global parameters appear in all channels while the local parameters appear only in the individual channel. Depending on the imaging modality, any fitting parameter θ (x, y, z/σPSF, photons, background) can be either linked as a global parameter among the channels or treated as a local parameter with different values in each channel. For global parameters, we define a transformation function to link parameters of different channels (translation and scale). The shared parameter in the ith channel can be written as: θpi = Spiθp + Δθpi. Here, Spi and Δθpi are the scaling and translation factor, respectively. In this work, Δθxi and Δθyi are defined as the shifts between the transformed ROI position and the actual ROI position which is rounded to integer pixels, as defined in item 4 of the section: Extraction of multi-channel single molecule data. The ratio of the photons between different channels SNi, used for fixed photon ratio fitting, is determined from experimental single molecule data as the mean of the detected photons per localization for each dye. Therefore, the first derivative for a global parameter θpi in the ith channel is given by: The first derivative for a local parameter θqj in the ith channel is 0 when i is not equal to j: Therefore, the Jacobian matrix can be defined as: The Hessian matrix is defined as: In the L-M algorithm, we updated the parameters by solving the linear equations: (H + λI)Δθ = J, with λ the damping factor and I a diagonal matrix equal to the diagonal elements of the Hessian matrix. The detailed algorithm can be found in Supplementary Note 4. Depending on different fitting modalities, we then calculate parameters of interest (e.g., color, polarization or z-position) from the fitted parameters (e.g., number of photons in each channel). Finally, we perform the usual post processing steps such as merging of localizations persisting over consecutive frames, drift correction and filtering based on log-likelihood and localization precision.
Simulation and analysis of multi-channel data with experimental PSF models
For biplane data simulation (Figure 1b and Supplementary Fig. 1), the biplane experimental PSF model of the SMLM challenge 2016 was used. For each single molecule, we use 5,000 detected photons and 40 background photons for the simulation. The photons were then split into two channels with 1:1 ratio. 1,000 single molecule images were simulated for each z position (range from -600 nm to 600 nm) on a ROI of 15×15 pixels. Only Poisson noise was added to the images. The simulated biplane single molecule data was then fitted with three different schemes: 1) global fit with x, y, z, photons and background photons shared; 2) global fit with x, y, and z parameters shared; 3) individual fit for each channel and combination of the parameters of different channels with CRLB weighted arithmetic mean (Supplementary Note 1). The localization accuracy was calculated as the root mean square error (RMSE) of the fitted coordinates compared to the ground truth.
For the ratiometric astigmatic simulation (Figure 1c, Supplementary Fig. 2, Supplementary Fig. 6 and Supplementary Fig. 7), a dual channel astigmatic experimental PSF acquired from multicolor beads was used. The photon distribution of these 4 different dyes were used for simulation: DY634, AF647, CF660C and CF680. The ratio of photons between two channels for the different dyes was determined from experimental data corresponding to Figure 2c as the mean of the detected photons per localization for each dye. We found photon ratios of I2/I1 = 0.39, 0.21, 0.07 and 0.02 for DY634, AF647, CF660C and CF680, respectively. Here, I1 and I2 are the photons from the bright and dark channels, respectively. For comparison of localization accuracy and CRLB (Supplementary Fig. 2), the photon ratio of 0.25 was used. 1,000 molecules with a ROI size of 15×15 pixels at each z position were used to calculate the RMSE. For ratiometric color separation (Supplementary Fig. 6 and Supplementary Fig. 7), 50,000 single molecules were randomly placed at axial positions between -600 nm and 600 nm. The photon distribution of each dye follows the distribution of the experimentally acquired single molecules (Supplementary Fig. 7a). Three different methods were used to determine the color information: 1) The dual channel data was fitted separately; 2) The dual channel data was fitted globally with x, y and z as global parameters, photons and background as local parameters; 3) The dual channel data was fitted globally with x, y, z and photons as global parameters, background photons as shared parameter. The ratio of the photons between different channels was fixed during fit. For the first two methods, the color discrimination was realized by thresholding the normalized photon ratio: (I1 − I2)/(I1 + I2). For the third method, the dual channel data was fitted with different fixed and pre-determined photon ratios of all 4 dyes between the two channels and we then chose the solution with the maximum likelihood.
For 4Pi single molecule data simulation (Supplementary Fig. 3), 2,000 photons/localization and 20 background photons/pixel were used for each objective. A full vectorial PSF model31 was used for simulations with the following parameters: NA 1.35, refractive index 1.40 (immersion medium and sample) and 1.518 (cover glass), emission wavelength 668 nm, astigmatism 100 mλ. 1,000 4Pi single molecule images with a ROI size of 15×15 pixels were simulated at each z position with four phase channels (0, π/2, π, 3π/2). The x and y positions are randomly distributed within -1 to 1 pixels around the center of each ROI. The simulated 4Pi single molecule data was then fitted with three different approaches: 1) global fit using IAB-based 4Pi-PSF model with x, y, z, phase, photons and background photons shared; 2) global fit using IAB-based 4Pi-PSF model with x, y, z and phase, parameters shared; 3) photometry based methods11.
GPU implementation of globLoc and speed evaluation
We implemented the globLoc fitter with both spline and Gaussian PSF model using CUDA C/C++ in NVIDIA CUDA®-enabled graphic cards. The framework of the L-M iterative fitting method follows the previous work21. Each thread is pointed to a multi-channel single-molecule data and performs the entire fitting process for each single molecule. We put the single-molecule data in the global memory of the GPU and employed 64 threads for each block for the computation. Both the CPU and GPU based C++ code were compiled in Microsoft Visual Studio 2019 and called via Matlab 2019a (Mathworks) MEX files. For speed evaluation (Supplementary Fig. 4), we ran the CPU code on a personal computer using an Intel Core i7-8700 processor clocked at 3.2 GHz with 16GB memory. For the GPU-based evaluation, an NIVDA GeForce GTX 3090 graphics card with 24.0 GB memory was used.
State-of-the-art workflows used for comparison
For biplane data analysis, we compared globLoc with the widely used ThunderSTORM software23. In the ThunderSTORM biplane analysis pipeline, a homography transformation is constructed from paired coordinates of the two channels. The biplane data is then fitted simultaneously using an astigmatic Gaussian PSF model. The detailed parameters used are shown in Supplementary Fig. 10 and Supplementary Table 1. For ratiometric multi-color assignment, we also compared to a workflow similar to that presented by Lehann et al (Supplementary Fig. 9)26. In short, localizations are fitted separately in the two channels. We then construct the transformation as described above and associate corresponding localizations in the two channels. Color assignment is then based on the relative fitted photon numbers.
Cell culture
Before seeding of cells, high-precision 24 mm round glass coverslips (No. 1.5H, catalog no. 117640, Marienfeld) were cleaned by placing them overnight in a methanol:hydrochloric acid (50:50) mixture while stirring. After that, the coverslips were repeatedly rinsed with water until they reached a neutral pH. They were then placed overnight into a laminar flow cell culture hood to dry them before finally irradiating the coverslips by ultraviolet light for 30 min.
Cells were seeded on clean glass coverslips 2 days before fixation to reach a confluency of about 50 – 70% on the day of fixation. They were grown in growth medium (DMEM (catalog no. 11880-02, Gibco)) containing 1× MEM NEAA (catalog no. 11140-035, Gibco), 1× GlutaMAX (catalog no. 35050-038, Gibco) and 10% (v/v) fetal bovine serum (catalog no. 10270-106, Gibco) for approximately 2 days at 37 °C and 5% CO2. Before further processing, the growth medium was aspirated, and samples were rinsed with PBS (RT) to remove dead cells and debris. Unless otherwise stated, all experimental replicates were performed on cells of a different passage with separated sample preparation.
Imaging Buffer
Glucose oxidase/catalase buffer supplemented with cysteamine (MEA) was used to image Nup96-SNAP-AF647-ELYS-CF660C-Nup62-DY634-WGA-CF680. GLOX+MEA contained 50 mM Tris/HCl pH8, 10 mM NaCl, 10% (w/v) D-glucose, 500 μg/ml glucose oxidase, 40 μg/ml glucose catalase and 35 mM MEA in H2O.
Preparation of four-color NPC samples
Cells (Nup96-SNAP-tag, catalog no. 300444, CLS Cell Line Service, Eppelheim, Germany) on glass coverslips were prefixed in 2.4% (w/v) FA in PBS for 20 s before incubating them 10 min in 0.5% (v/v) Triton X-100 in PBS. Fixation was completed in 2.4% (w/v) FA in PBS for 20 min. FA was quenched for 5 min in 100 mM NH4Cl in PBS and then washed 3x 5 min in PBS. Fixed cells were blocked with Image-IT signal enhancer for 30 min and then incubated with 1 μM BG-AF647, 0.5% BSA and 1 mM DTT in PBS for 1 h to stain Nup96-SNAP-tag. Cells were washed 3x for 5 min with PBS and subsequently blocked with 5% (v/v) NGS (catalog no. PCN5000, lifeTech) in PBS for 1 h. Primary antibody labeling against ELYS was achieved by incubation with anti-AHCTF1 (HPA031658, Sigma-Aldrich) diluted 1:40 in 5% (v/v) NGS in PBS for 1 h. Coverslips were washed 3 times for 5 min with PBS to remove unbound antibody and subsequently stained with CF660C labeled anti-rabbit antibody (20183, Biotium, Fremont, CA) diluted 1:150 in PBS containing 5% (v/v) NGS for 1 h. After 3 washes with PBS for 5 min, the sample was postfixed for 30 min using 2.4% (w/v) FA in PBS, rinsed with PBS, quenched in 50 mM NH4Cl for 5 min and rinsed 3x 5 min with PBS. Labelling against Nup62 was performed by incubation with mouse anti-Nup-62 primary antibody (610498, BD Bioscience) diluted 1:50 in 5% NGS/PBS for 2h, 3x 5min washes of the coverslips with PBS and incubation over night at 4degC with 1:150 diluted secondary anti-mouse-DY634 antibody in 5%NGS/PBS. Unbound antibody was removed from the sample by washing 5 times with PBS. All incubations except otherwise stated were carried out at RT. Buffers used were also pre-equilibrated to RT.
Shortly before imaging, the sample was incubated for 10 min with 1:5000 diluted WGA-CF680 (29029-1, Biotium, Fremont, CA) in 100mM Tris, pH 8.0, 40mM NaCl, rinsed 3x with PBS and mounted onto a custom manufactured sample holder in imaging buffer. The holder was sealed with parafilm.
Preparation of DY634-labelled secondary anti-mouse antibody
50ul of donkey anti-mouse IgG (H+L) (1,3mg/ml) (715-005-151, Dianova) was incubated with a 10-fold molar excess of DY634-NHS (634-01, Dyomics) in a final volume of 100ul PBS pH 7,4 overnight at RT. The labelled antibody was purified from free dye by running over an PBS equilibrated Zeba Spin desalting column (89889, Thermo Scientific) by gravity flow. Fractions containing the peak of the labelled antibody were identified by SDS-PAGE and pooled.
Microscope setup
Single-objective SMLM image acquisition was performed at room temperature (RT, 24 °C) on a custom built microscope equipped with a high NA oil immersion objective (160x, 1.43-NA oil immersion, Leica, Wetzlar, Germany) described previously32. A commercial laser box (LightHub®, Omicron-Laserage Laserprodukte, Dudenhofen, Germany) equipped with Luxx 405, 488 and 638, Cobolt 561 lasers and an additional 640 nm booster laser (iBeam Smart, Toptica, Munich, Germany) were combined for wide field illumination. Lasers were focused onto a speckle reducer (LSR-3005-17S-VIS, Optotune, Dietikon, Switzerland) and coupled into a multi-mode fiber (M105L02S-A, Thorlabs, Newton, NJ, USA). The lasers were triggered using an FPGA (Mojo, Embedded Micro, Denver, CO, USA) allowing microsecond pulsing control of lasers. The output of the fiber was magnified by an achromatic lens and imaged into the sample plane. A laser clean-up filter (390/482/563/640 HC Quad, AHF, Tübingen, Germany) was placed in the excitation beam path to remove the fluorescence generated by the fiber. The focus of microscope was stabilized by a 785 nm infrared laser (iBeam Smart, Toptica, Munich, Germany) that was projected through the objective and reflected by the coverslip onto a quadrant photodiode, which was used as closed-loop feedback signal to the objective piezo stage (P-726 PIFOC, Physik Instrument, Karlsruhe, Germany). The fluorescence emission was filtered by a bandpass filter 676/37 (catalog no. FF01-676/37-25, Semrock) and then split into two channels (separated by ∼ 400 nm axially) using a 50:50 beamsplitter for biplane imaging with AF647. The astigmatic 3D imaging was acquired using a cylindrical lens (f = 1,000 mm; catalog no. LJ1516L1-A, Thorlabs) to introduce astigmatism. For astigmatic multicolor imaging with DY 634, AF647, CF660C and CF680, the fluorescence of the ratiometric multi-color imaging was split by a 665 nm long pass dichroic (catalog no. ET665lp, Chroma), filtered by a 685/70 (catalog no. ET685/70m, Chroma) bandpass filter for the transmitted light and a 676/37 (catalog no. FF01-676/37-25, Semrock) bandpass filter for the reflected light. An EMCCD camera (Evolve512D, Photometrics, Tucson, AZ, USA) was used to collect final fluorescence. Typically, we acquire 100,000 – 300,000 frames with 30 ms exposure time and laser power densities of ∼ 15 kW/cm2. The pulse length of the 405 nm laser is automatically adjusted to retain a constant number of localizations per frame.
4Pi-SMLM image acquisition was performed at RT based on an instrument as described previously11 with minor modifications. Two magnification matched silicone immersion objectives (1.35 NA, UPLSAPO, 100XS, Olympus) were used for better refractive index matching. The system was equipped with four excitation lasers: 405 nm (IBEAM-SMART-405 nm, 150 mW, Toptica), 488 nm (IBEAM-SMART-488-S-HP, 200 mW, Toptica), 560 nm (2RU-VFL-P-1500-560-B1R, MPB Communications, Pointe-Claire, Canada) and 642 nm (2RU-VFL-P-2000-642-B1R, MPB Communications). The excitation laser was filtered by a clean-up filter (390/482/563/640 HC Quad, AHF) and then reflected by a quadband dichroic (405/488/561/635, F73-867, AHF). The emission fluorescence was passed through the dichroic and then filtered by a quadband filter (432/515/595/730 HC, F67-432, AHF). The fluorescence was additionally filtered by a bandpass filter 676/37 (catalog no. FF01-676/37-25, Semrock) before collection on an sCMOS camera (ORCA-Flash 4.0v2, Hamamatsu). ∼ 200,000 images were acquired with 25 ms exposure time. The pulse length of the 405 nm laser is automatically adjusted to retain a constant number of localizations per frame.
Data availability statement
The experimental biplane and 4 color 3D astigmatic datasets can be freely downloaded from this website: https://www.embl.de/download/ries/globLoc/. All other data are available upon reasonable request from the corresponding authors.
Code availability
Source code for the software used in this can be freely downloaded at https://github.com/jries/SMAP/tree/develop/fit3Dcspline/GlobLoc.
Author contributions
Y.L. and J.R. conceived the approach, developed the methods, wrote the software, and analyzed the data. W.S. and D.W. analyzed the biplane data and wrote the tutorial for globLoc. S.L. acquired and analyzed the 4Pi data. U.M. and Y.L. acquired the data. Y.L. and J.R. wrote the manuscript with input from all authors.
Competing financial interests
The authors declare no competing financial interests.
Supplementary materials
Supplementary Note 1
Parameter Merging for individual fits of multi-channels
To merge parameters returned from individual fits of different channels, weighted arithmetic mean of parameters from all channels was used. Here, we used the reciprocal of the estimated CRLB as the weights for each parameter: Here, θmi is the set of parameters being estimated in the ith channel and is the corresponding CRLB. The significance of this choice is that this weighted mean is the maximum likelihood estimator of parameters of different channels under the assumption that they are independent and normally distributed with the same mean. Therefore, this combination could return the optimized localization precision. As show in Supplementary Fig. 1a and Supplementary Fig. 2a, the localization accuracy of the global fit with linking xyz is similar to that of the CRLB-weighted individual fits. However, the global fit could substantially improve the localization accuracy in z by additionally linking photon parameters.
Supplementary Note 2
Calculation of multichannel CRLB
To quantify the localization precision of globLoc fitter, we compared it with the CRLB which is the limiting lower bound of the variance for any unbiased estimator. The general definition of CRLB is evaluated as the diagonal element of the inverse of the Fisher information matrix: where is the variance of an estimator and I(θ) is the Fisher information matrix. Depending on how the parameters are linked during fit, the Fisher information matrix is defined as Here, is defined as Equation (1) in Methods, μki is the expected photon number in the kth pixel of the ith channel. θp is the set of global parameter and θqi is the set of local parameters of ith channel.
Supplementary Note 3
Derivatives for IAB-based 4Pi-PSF model
The IAB-based 4Pi-PSF model is written as in ref. 25: Here, φ is the interference phase. I(x, y, z), A(x, y, z) and B(x, y, z) are phase independent and slowly varying real functions of x, y, z. In order to construct the Hessian and Jacobian matrix, the following partial derivatives of x, y, z, phase, photons and background are used: Here, the cubic splines are used to interpolate the 3D matrices I, A and B to calculate the partial derivative along x, y and z, separately21,30. The L-M iterative process is considered to be converged when the ratio of the relative change of is less than 10−6 compared to the last iteration.
Supplementary Note 4
Method for L-M nonlinear optimization of multichannel data
For maximum likelihood estimation, the cost function is defined as Equation (1) in Methods. During the L-M optimization process, Hessian (Hm,n) and Jacobian (Jm) matrix are defined as Equation (4) and (5) in the Methods. The detailed optimization algorithm used in this work is described below:
Calculate (old) with a user defined starting parameters θm.
Initialize λ as 0.1. λ is a damping factor that controls whether the L-M fit should behave more as a gradient descent fit method (lambda<<1) or an expansion fit method (lambda>>1).
Calculate the updates of each parameter Δθm by solving the linear equations:
Derive new trial-fit-parameters θm(new) as follows: Here, is a clamping factor which controls the maximum change of parameter θm during each iteration. If the sign of Δθm has changed since last update, is multiplied by 0.5 to suppress oscillations during optimization and damp excessively large corrections33.
Determine (new) with the new parameter using Equation (1) in the Methods.
If , substitute λ with λ/10, and θm(old) with θm(new) and continue with step 3.
If , substitute λ with 10λ, keep θm(old) unchanged, and continue with step 3.
If , keep λ unchanged, substitute θm(old) with θm(new), and continue with step 3.
This iterative calculation may be stopped when one of the following conditions is met:
The ratio of the relative change of and decreases to less than a specified value (10−6 in this work).
The iteration time of the calculation loops exceeds a maximum number.
Acknowledgements
We thank L. Zhou, S. Fu, J. Chen and M. Li for testing of the tutorials of globLoc. This work was supported by the Guangdong Natural Science Foundation Joint Fund (2020A1515110380 to Y. L.), Shenzhen Science and Technology Innovation Commission (Grant No. KQTD20200820113012029), the Startup grant from Southern University of Science and Technology, the European Research Council (grant no. ERC CoG-724489 to J.R.), the National Institutes of Health Common Fund 4D Nucleome Program (grant no. U01 EB021223 to J.R.), the Human Frontier Science Program (RGY0065/2017 to J.R.), the EMBL Interdisciplinary Postdoc Programme (EIPOD) under Marie Curie Actions COFUND (Y.L.), and the European Molecular Biology Laboratory.