Abstract
Mean-Shift Super Resolution (MSSR) is a principle based on the Mean Shift theory that extends spatial resolution in fluorescence images, beyond the diffraction limit. MSSR works on low- and high-density fluorophore images, is not limited by the architecture of the detector (EM-CCD, sCMOS, or photomultiplier-based laser scanning systems) and is applicable to single images as well as temporal series. The theoretical limit of spatial resolution, based on optimized real-world imaging conditions and analysis of temporal image series, has been measured to be 40 nm. Furthermore, MSSR has denoising capabilities that outperform other analytical super resolution image approaches. Altogether, MSSR is a powerful, flexible, and generic tool for multidimensional and live cell imaging applications.
Introduction
Super-resolution Microscopy (SRM), which encompasses a collection of methods that circumvent Abbe’s optical resolution limit, has dramatically increased our capability to visualize the architecture of cells and tissues at the molecular level. There are several approaches to SRM which vary in terms of the final attainable spatial and temporal resolution, photon efficiency, as well as in their capacity to image live or fixed samples at depth [1, 2]. One class of techniques exceed the diffraction limit by engineering the illumination or the point spread function (PSF), such as SIM and STED [3–5]. These techniques can be used for live imaging although they require specialized hardware and dedicated personnel for maintenance and operation. Single-molecule localization methods (e.g., STORM, PAINT, PALM) [6–9] that localize individual emitters with nanometer precision require temporal analysis of several hundred-to-thousands of images and are prone to error due to fast molecular dynamics within live specimens.
Some SRM computational methods have few or no demands on hardware or sample preparation and provide resolution improvements beyond the diffraction limit, i.e., fluorescence fluctuation-based super resolution microscopy (FF-SRM) approaches [10–13]. The quantity and performance of computational methods have both increased over the past decade given the many advantages they present, such as their low barriers to entry and generic applicability to data acquired with any microscopy modality (wide-field, confocal, or light-sheet). However, these methods also present some limitations, such as the possible introduction of artifacts [14], the requirement for high signal-to-noise ratio (SNR) data and the acquisition of tens to hundreds of frames [10–13], which limit their applicability to reconstruct fast dynamical processes.
The problem of spatial resolution in optical microscopy can be addressed from the statistical point of view. In the case of fluorescence microscopy, the process of photon emission from punctual sources (fluorescence emitters) can be considered as a discrete distribution of information, where the unitary element of the distribution is the photon [15]. In this scenario, the problem of spatial resolution gets reduced to the problem of finding modes of information, regardless of the shape of the distribution, hence, disconnecting the problem of optical resolution from the diffraction boundary [16].
Here, we introduce the Mean Shift Super-Resolution principle for digital images ‘MSSR’ (pronounced as messer), derived from the Mean Shift (MS) theory [17, 18].
MSSR extends the resolution of any single fluorescence image up to 1.6 times, including its use as a resolution enhancement complement after the application of other super-resolution methods.
By computing the local magnitude of the Mean Shift vector, MSSR generates a probability distribution of fluorescence estimates whose local magnitude peaks at the source of information. As a result of that, the spatial distribution becomes ‘refined’ (i.e., for a Gaussian distribution of fluorescence its width shrinks). Additionally, we demonstrate the extended-, enhanced- and super-resolving capabilities of MSSR as a standalone method for a variety of fluorescence microscopy applications, through a single-frame and temporal stack analysis, allowing resolution improvements towards a limit of 40 nm.
Open-source implementations of MSSR are provided for ImageJ (as a plugin), R, and MATLAB, some of which take advantage of the parallel computing capabilities of regular desktop computers (Supplementary Note 7). The method operates almost free of parameters; users only need to provide an estimate of the point spread function (PSF, in pixels) of the optical system, choose the MSSR order, and decide whether a temporal analysis will take place (Supplementary Material and MSSR Manual). The provided open-source implementations of MSSR represent a novel user-friendly alternative for the bioimaging community for unveiling life at its nanoscopic level.
Results
The MSSR principle
MSSR is tailored around the assumption that fluorescence images are formed by discrete signals collected (photons) from point sources (fluorophores) convolved with the PSF of the microscope (Supplementary Notes 1, 2 and 3). Processing a single image with MSSR starts with the calculation of the MS, which guarantees that large intensity values on the diffraction-limited (DL) image coincide with large positive values in the MSSR image (Supplementary Note 4). Further algebraic transformations then restore the raw intensity distribution and remove possible artifacts caused by the previous step (edge effects and noise dependent artifacts), giving rise to an image that contains centers of density with a narrower full width at half maximum (FWHM) (Figure 1a). This procedure is denoted by MSSR of order zero (MSSR0), and it is the first stage which shrinks emitter distribution.
The MS is locally computed by a kernel window that slides throughout the entire image, subtracts the sample mean (weighted local mean) as well as the central value of the kernel using a spatial-range neighborhood (Supplementary Notes 2 and 3, Supplementary Figure S5, Supplementary Table S1) [17, 18]. The MS is a vector that always points towards the direction of the intensity gradient and its length provides a local measure of the fluorescence density and brightness [19–21]; its magnitude depends on the value difference between the central pixel of the neighborhood and the surrounding pixels. A mathematical proof, provided in Supplementary Note 4, demonstrates that the minimum MS value, computed from a Gaussian distribution, matches with the point of maximum intensity of the initial distribution (Supplementary Note 4, Supplementary Figure S6).
The increase in resolution offered by MSSR0 was evaluated by the Rayleigh and Sparrow limits [22–24], which are two criteria that establish resolution bounds for two near-point sources (Figure 1b). Processing with MSSR0 of two-point sources located at their resolution limit (2.5 σ and 2 σ for Rayleigh and Sparrow limit respectively, Figure 1c vertical discontinuous lines) decreases the dip (height at the middle point) [25] within their intensity distributions (Figure 1b and 1c). Processing a single image with
MSSR0 shifts the resolution limit by 26 % and 20 % according to the Rayleigh and Sparrow limits, respectively, and reduces the FWHM of individual emitters (Figure 1c vertical continuous lines). A comparison of the shrinkability of MSSR 0 applied to Gaussian and Bessel distributions are shown in Supplementary Figure S9. The reduction of FWHM of Bessel distribution at different wavelengths of the visible spectrum are shown in Supplementary Figure S10.
Since the result of MSSR0 is an image, we used the resulting image to seed an iterative process (Figure 2a). We refer to this as higher-order MSSR (MSSRn, with n>0), which delivers a further gain of resolution per n-iteration step (Figure 2a and Supplementary Figure S11). As the order of MSSRn increases, both the FWHM of emitters (Supplementary Figure S12) and the dip of their intensity distribution decrease (Figure 2b). Numerical approximations indicate that two point-sources separated at 1.6 σ are resolvable with MSSR3, but not when their separation is 1.5 σ (Figure 2b). The separation of 1.6 σ sets the theoretical resolution limit of MSSRn.
In summary, MSSRn processing extends the spatial resolution of single DL images. The procedure of applying MSSRn to a single DL image will be defined as sf-MSSRn.
MSSR is a deconvolution approach at the nano scales
In optical microscopy, objects significantly closer to the diffraction boundary can be resolved with clever illumination and detection schemes (i.e., SIM, Airy Scan, 4 Pi, I5M, STED, SMLM, etc.) [3, 26–28], or by careful image analysis, reviewed in [29]. Rayleigh criterion is a bit too conservative, in the sense that achieving a decrease of the Dip formed by the joint distribution shaped by two adjacent emitters might be interpreted as surpassing the diffraction boundary (Supplementary Figure S13 a, b).
Repeating the same procedure using a joint distribution shaped by two adjacent emitters located at the Sparrow limit there is no further gain of resolution, as the dip remains constant, taking the value of 1 (Supplementary Figure S13 e, f).
MSSR aims to revert the effect of diffraction on optical microscopy, so it can be considered as a deconvolution process. As the diffraction can be modeled with a Gaussian spread, the pixel value is a superimposition of spreads from individual emitters (Supplementary Notes 1, 2 and 3). The goal is to reduce the spread. The latter can be accomplished by “sharpening by blur” [30, 31]. In MSSR, the computation of the MS is the blurring process used to sharpen the image. What makes sf-MSSRn unique is the fact that it extends spatial information down below the Sparrow limit. Processing the joint distributions of Supplementary Figure S13 with sf-MSSRn of any order n leads to a decrease of the dip value (Supplementary Figure S13 c, d, g, f). Furthermore, sf-MSSR3 processing collapses the dip to zero for both Rayleigh and Sparrow conditions (Supplementary Figure S13 d, h).
To illustrate how MSSR works by sharpening features down the diffraction barrier, we provide comparative data against: Wiener deconvolution [32], Richardson-Lucy deconvolution [33, 34] and the Radiality Maps (RMs) [11].
Supplementary Figure S14 shows that Wiener deconvolution partially restores the effect of diffraction, but without a dramatic increase in spatial resolution. Interestingly, Richardson-Lucy deconvolution provides a noticeable increase in resolution at the boundaries of the Rayleigh limit but fails to extend spatial resolution down below the Sparrow limit (Supplementary Figure S14).
Gustafsson et al. showed that the RMs of SRRF provide a resolution increase down to 0.7 times the Gaussian FWHM [11], where when the peak separation between isolated is greater than 0.7 times the FWHM of the PSF, they can be directly resolved without further enhancement provided by higher-order statistical analysis. MSSR0 and the Radiality Maps are similar in the sense that both perform sharpening and smoothing. Supplementary Figure S14a shows that both MSSR and the RMs overcome the Rayleigh diffraction limit [22]. However, the RMs produce undesired artifacts which are absent when using sf-MSSR0 (Supplementary Figure S14b), which is in agreement with the reported spatial artifacts introduced [35]. Figures 1, 2 and Supplementary Figure S14 show that sf-MSSR0 reliably provides spatial resolution gains, free of image analysis artifacts, down to the Rayleigh limit, hence, allowing the study of nanoscopic regimes at the boundaries of the Sparrow limit.
With such data in hand, it is concluded that sf-MSSRn is a deconvolution process that extends spatial information of DL images at the nano scales.
MSSR extends spatial resolution in fluorescence microscopy images
To empirically test the ability of MSSR to extend spatial resolution within a single DL image, a commercial nanoruler sample (GATTA-SIM140B, GATTAquant) was imaged by Structured Illumination Microscopy (SIM) and Total Internal Reflection Fluorescence (TIRF) microscopy, which was then processed by sf-MSSRn. The iterative processing of the widefield data with sf-MSSR3 reveals the two fluorescence emitters located at a separation of 140 nm, which is consistent with the result obtained by SIM (Figure 2c).
A theoretical approximation of the Rayleigh limit for a GATTA-SIM 140B nanoruler (Figure 2c of the main manuscript) with λem= 525 nm and NA = 1.4, is d = (0.61*525)/1.4 = 229 nm (Table 1). Figures 1c and 2b show that MSSR processing theoretically extends spatial resolution. Computation of resolution on sf-MSSR0 using the Rayleigh criterion (at 0.74 of the Dip) gives a spatial resolution of 160 nm, which corresponds to a resolution change to 0.69 times (0.69x) the resolution limit. Using higher orders of sf-MSSRn with n = 1, 2, 3, gives a resolution change of 0.66-0.64x the resolution limit (Table 1). Table 1 also shows the spatial resolution measured on the GATTA-SIM 140B nanorulers (experimental data), computed through decorrelation [36], using the Image Decorrelation plug-in for Fiji/ImageJ. Decorrelation computes the maximal observable frequency in an image (K0) as a proxy of spatial resolution (Resolution = 1/ K0). Note that sf-MSSRn dramatically reduces resolution as a function of the n-order.
To further test the attainable resolution by sf-MSSRn we used the ArgoLight test slide, acquiring images of the pattern formed by gradually spaced lines of fluorescent molecules (Argo-SIM, pattern E). The distance between lines increases from 0 to 390 nm with a step change of 30 nm: 0 nm, 30 nm, 60 nm, etc. Figure 3 shows the application of sf-MSSRn to the confocal microscopy images of the Argo-SIM micropattern. As expected, the confocal acquisition allows to resolve parallel rows of fluorophores located at 240 nm. Remarkably, sf-MSSR0 processing extended spatial resolution down to 0.5 times the confocal resolution, allowing to discriminate parallel rows of fluorophores located at 120 nm. It is worth mentioning that this improvement comes at zero hardware cost, compared to other methods requiring specific optics/detectors such as the Airyscan [37] and Re-scan confocal microscopy [38].
Higher orders of sf-MSSRn create a saddle point between parallel rows of emitters located in the range of 60 - 90 nm at the boundaries of the Rayleigh limit (Figure 3b, c).
MSSR enhances the resolution of images with extended resolution
Based on the MSSR capabilities to generate a micrography with extended spatial resolution after processing a single fluorescence image, we explored if a pre-existing image with extended resolution can be further enhanced by sf-MSSRn.
The Argo SIM micropattern was imaged using an Airyscan detector with other experimental settings as in Figure 3. Images were further deconvolved with the corresponding Airyscan algorithm [39]. Figure 4 shows the application of sf-MSSRn to the Airyscan microscopy images of the Argo-SIM micropattern. Within the Airyscan processed image it is possible to resolve parallel rows of fluorophores located at 180 nm, but not 120 nm or less. sf-MSSR0 processing enhanced the spatial resolution of the same Airyscan data, allowing to discriminate parallel rows of fluorophores located at 120 nm, or less (Figure 4 a, b). Higher orders of sf-MSSRn create a saddle point between parallel rows of emitters located in the range of 60 - 120 nm (Figure 4 b, c).
Remarkably, compared with the confocal original data in Figure 3, where the last resolvable line pair is the one corresponding to a distance of 240 nm, the value of 120 nm obtained in Figure 4 by applying sf-MSSR0 to Airyscan processed data corresponds to a 2-fold improvement in resolution. The first reported applications of Airyscan technology allowed an improvement in resolution of 1.7X [40], while only following protocols claim that a 2-fold improvement might be achieved [39], compared to standard confocal detection.
We then applied sf-MSSR0 on a SIM image of sister meiotic chromatids of mouse chromosomes [41]. Similar to the results obtained above with Airyscan, Figure 4d shows that processing with SIM images with sf-MSSR0 enhances both the contrast and resolution of the final image.
MSSR enhances the resolution of super-resolved images
We explored if a pre-existing super-resolved image can be further enhanced by sf-MSSR.
First, we used a temporal stack of DL images of tubulin-labeled microtubules collected at high fluorophore density [42] (previously used to test and compare a variety of SRM algorithms) [43], which were subject to FF-SRM analysis [10, 44]. ESI, SRRF or MUSICAL were used to compute a single enhanced-resolved image (Figure 5a) [11–13]. Supplementary Note 9 contains an in-depth comparison of sf-MSSR0 reconstructions combined with either ESI, SRRF and MUSICAL, which achieve super-resolution through a temporal analysis [11–14]. Post-processing of ESI, SRRF or MUSICAL images with sf-MSSR0 enhances contrast and spatial resolution (Figure 5a).
Second, a sequence of images of randomly blinking emitters placed along a synthetic tubular structure [45] was processed with sf-MSSR0 after analysis with MUSICAL. In both reconstructions, three regions (small squares in Figure 5b) were chosen to assess the gain in resolution, visualized in terms of the distance between the normalized intensity distributions peaks. MSSR further resolves the edges of the synthetic structures on the MUSICAL-processed image without changing the position of the distribution peaks (Figure 4c) as predicted by our theory (Supplementary Notes 1 to 4).
Lastly, we set up an experimental assay to examine the achievable resolution by MSSR alone, or in combination with either confocal or stimulated emission depletion (STED) microscopy. Figure 6 a, b, show that STED, but not confocal imaging allows to discern chromosomal territories within the chromatin of 2-cell stage embryo, as observed by the lack of colocalization of acetylated (ac, i.e., ‘active’) or methylated (me, i.e., ‘inactive’) chromatin states, H3K27me3, and H3K27ac respectively [46].
Remarkably, Figure 6c shows that sf-MSSR1 processing of the confocal image allows to reach a similar experimental conclusion, that there is no colocalization between H3K27me3 and H3K27ac signals.
Figure 6 also shows the result of post-processing of the STED image with sf-MSSR1 (Figure 6g). We note an increase in contrast and resolution from confocal to STED and then to STED + sf-MSSR1 (compare panels e, f, and g in Figure 6). To assess the resolution change provided by either STED or STED + sf-MSSR1 we used the image decorrelation approach [36]. The bottom panels of Figure 6 e, f, and g, show the resolution computed at a confocal image (315 – 330 nm), STED (80 – 100 nm) and STED + sf-MSSR1 (19 – 22 nm). With these results, we conclude that sf-MSSR1 processing of STED data provides a further increase in spatial resolution.
MSSR achieves super resolution by analyzing fluorescence intermittency over time
Analyzing the temporal dynamics of fluorescence is central to achieving the highest attainable gain of resolution by FF-SRM approaches (reviewed in [36]). FF-SRM methods analyze higher order temporal statistics looking to discern correlated temporal information (due to fluorescent fluctuations) from uncorrelated noise (i.e., noise detector). In theory, MSSR can be applied to a sequence of images (Supplementary Note 5). Based on the increase in resolution offered by FF-SRM approaches (SRRF, MUSICAL), we investigated whether a further resolution gain could be achieved by applying a temporal analysis to a sequence of single-frame MSSR images (denoted by t-MSSRn) (Figure 7a). Pixel-wise temporal functions (PTF), such as average (Mean), variance (Var), the temporal product mean (TPM), coefficient of variation (CV) or auto-cumulant function of orders 2 to 4 (SOFI2, SOFI3, SOFI4) [10], can be used to create an image with enhanced spatial resolution (Supplementary Note 5.2, Supplementary Table S2).
To experimentally validate the increase in resolution by both sf-MSSRn and t-MSSRn we used two different nanoruler systems, an in-lab CRISPR/dCas12a nanoruler, used to score nanoscopic distances between individual fluorescent sites down to 100 nm, and a commercial nanoruler with fluorophores positioned at 40 nm of separation (GATTA-PAINT, 40G, and 40RY. Gattaquant).
The CRISPR/dCas12a nanoruler system consists of a dsDNA with four binding sites for dCas12a uniformly distributed every 297 bp (equivalent to ∼ 100 nm of separation) (Supplementary Figure S42a). To validate this system, we imaged the association of the CRISPR/dCas12a complex to the binding sites on the dsDNA by atomic force microscopy (AFM) and measured the distance between each dCas12a complex (Supplementary Figure S42b).
The CRISPR-dCas12a nanorulers were then imaged by TIRF microscopy for further MSSR analysis. We used a DNA-PAINT approach for fluorescence indirect tagging [47], in which a fluorescent ssDNA probe hybridizes with an extension of the gRNA. The “blinking” of the fluorescence signal is attained by events of association and dissociation between the fluorescent probe and the gRNA on the CRISPR/dCas12a nanoruler at the binding site.
In the DL image, amorphous spot-like fluorescent patterns were observed (Figure 7b). sf-MSSR3 processing of either an isolated frame or an average projection of the corresponding stack of 100 images (DL-AVG) could not resolve individual CRISPR/dCas12a binding sites (Figure 6b), and only after processing by t-MSSR3 did individual binding sites became resolved (Figure 7c). The result of t-MSSR3 varied in relation to the temporal function used (Figure 7c). The best result for this nanoruler was obtained by the pixel-wise temporal variance (Var) of the sf-MSSR3 stack (Figure 6c). t-MSSR3-Var resolved nearby emitters engineered to recognize binding sites located at 100 nm (Supplementary Movie S1), provided by scoring association-dissociation events between the imaging probe and the gRNA.
Once the CRISPR/dCas12a nanorulers were super-resolved by t-MSSR3-Var (Figure 7c), we scanned all the individual emitters in close proximity to determine the average distances between different dCas12a associated to the dsDNA binding sites (theoretically interspaced by 100 nm, Supplementary Figure S42). Since DNA in solution is a semi-flexible polymer [48], the measured distances resulted in three different distributions: 91 ± 31 nm, 220 ± 52 nm, 323 ± 19 nm (Figure 7d), which accounts for dCas12a separated by one, two and three binding sites along the dsDNA, respectively. These results confirm that t-MSSR3 can successfully resolve nanoscopic distances.
To explore the resolution limit attainable by t-MSSRn even further, we looked at a nanoruler system with smaller separation between fluorophore sites (from Gattaquant) (Supplementary Figure S43a). Figure 7e shows that using the TRA or TRM pixel-wise temporal functions in combination with MSSRn (t-MSSRn) does not provide extra resolution enhancement. TRA provides t-MSSR3 reconstructions less influenced by noise (recommended in imaging scenarios of marginal signal-to-noise ratios) and TRM delivers reconstructions whose intensity distribution is less dominated by constantly emitting sources, i.e., fiducial markers. In stark contrast, higher spatial resolution regimes, ∼ 0.15 times the FWHM of the PSF (PSF σ: [0.4 - 0.5]), are achieved by t-MSSRn when encompassing pixel-wise temporal functions of higher-order statistics such as Variance (Var), Coefficient of Variation (CV), or SOFI (TRAC2-4). Analysis with t-MSSR3 of 300 DL images using either Var or CV revealed individual fluorescent spots at 40 nm apart (Figure 7e and Supplementary Figure S43b). The data presented in Figure 7e demonstrate that t-MSSR3 resolves nanoscopic distances at 40 nm, validating a lower experimental spatial resolution bound of 0.5 σ (≈ 40 nm), which depends on the emission wavelength of the fluorophore (Figure 7e, Supplementary Figure S10c). In comparison, SRRF and MUSICAL were not able to resolve fluorescent emitters located 40 nm apart, consistent with their limit within the range of 50–70 nm (Figure 7f) [11–13].
Analyzing the temporal dynamics of fluorescence intermittency is central for any FF-SRM approach [10–13]. The highest attainable spatial resolution is influenced by blinking statistics, i.e., photokinetics for fixed fluorophores [11-13,49], and binding energies for diffusible fluorescent probes [50]. These factors, including the density of fluorophores emitting per frame, impinge on the final resolution of the reconstruction. Supplementary Figure S44 shows the pixel-wise temporal dynamics of two GATTA paint ranorulers (ATTO 655), where the temporal dynamics of a given nanoruler containing three binding sites can be studied by analyzing the temporal fluorescence fluctuations at nearby pixels. Note that diffraction imposes constraints to distinguish the temporal dynamics at individual PAINT binding sites. Noteworthy, sf-MSSR3 processing allows to untangle, in space and in time, the fluorescence dynamics at individual PAINT binding sites, where the fluorescence signal peaks due to the transient binding of a fluorescent labeled DNA strand to its corresponding binding site within the nanoruler.
Single frame nanoscopy, free of noise-dependent artifacts
The theory of image processing by MSSR (Supplementary Note 5), suggests that it should be robust over a wide range of SNR, granted by four factors. First, when processing a single frame, MS works as a local spatial frequency filter (a smoothing filter); regions corresponding to the image background are homogenized by the kernel window, reducing variation in background noise. Second, one of the steps of the MSSR procedure is to remove the MS negative constraints. The goal of this step is to remove an artifact caused by the MS calculation itself when applied to Gaussian and Bessel distributions. When calculating the complement of the resulting distribution, a valley (or depression) is generated between the peak intensity and the tails, which lies in the negative values and is referred to as negative constraints. After this, the artifact is removed. Third, when using a PTF, nanoscopic information is enriched due to temporal oversampling of the hidden fluorescent structure. Fourth, the spatial kernel of the MSSR algorithm operates within the subpixel realm; the number of neighboring pixels is digitally increased through interpolation (i.e., bicubic interpolation [51] for single frame analysis and Fourier interpolation [52] for temporal analysis, Supplementary Note 6) providing digital oversampling of the emitters’ locations (Supplementary Note 6).
We then experimentally assessed the capacity of MSSR to denoise fluorescence images and determine whether it introduces noise-related artifacts. We used a PSFcheck slide [53], which contains an array of regular fluorescent nanoscopic patterns shaped by laser lithography (Figure 8). Analysis with sf-MSSRn or t-MSSRn showed, in comparison to alternative approaches, striking denoising capabilities without introducing noticeable artifacts (Figure 8a, Supplementary Note 9). These artifacts, resembling amorphous nanoscopic structures around the fluorescent ring or within it, were commonly found at reconstructions generated by other analytical techniques (Supplementary Figure S29).
Starting at a SNR > 2, sf-MSSR1 provides reliable SRM reconstructions of comparable quality to other SRM approaches, which demand the temporal analysis of the fluorescence dynamics (Figure 8a and Supplementary Note 9). We quantified the quality of the reconstructions by calculating the Resolution Scaled Pearson (RSP) coefficient and the Resolution Scaled Error (RSE), which provide a global measurement of the quality of the reconstruction by comparing the super-resolution image and the reference image (in this case, the DL image) [14]. Higher RSP and lower SRE values are associated with reliable reconstructions (Supplementary Note 8). When the SNR is above 5, all tested algorithms perform similarly well in quality (Figure 8b), but their global errors differ from each other (Figure 8c). As expected, the RSE increased as a function of the SNR of the input images for any tested algorithm (Figure 8c).
The performance of MSSR in achieving a satisfactory reconstruction was assessed by varying the number of input images using a temporal analysis scheme (Supplementary Note 8). With SNR > 2 input data, RSP reaches near maxima values and RSE near minima values when processing a single frame (Supplementary Figure S28, Supplementary Movie S2). However, when computing MSSR using low SNR input data (SNR ∼ 2) a temporal analysis is required as RSP and RSE values reach a plateau only when a temporal stack of as few as 25 images is used (Supplementary Movie S3). These findings illustrate that the minimal number of frames needed by MSSR to provide a reliable reconstruction depends on the information itself, i.e., on the SNR and on the fluorescence blinking statistics of the specimen (Supplementary Movies S1-S3); and can be determined by computing RSP and RSE as function of the number of analyzed frames with t-MSSRn (Supplementary Figure S28).
Nanoscopic resolution with conventional fluorescence imaging
MSSR addresses the problem of spatial resolution in optical fluorescence microscopy from the statistical point of view, where the process of photon emission from punctual sources is considered as a discrete distribution of information, provided by detected photons. Enhanced or extended resolution is achieved by finding local modes of information without taking any prior about the local shape of the distribution. The latter makes the MSSR principle compatible with any imaging approach where the distribution of information is of discrete nature (i.e., photons), and emanates from discrete sources (i.e., fluorophores).
To showcase the versatility of MSSR to extended-, enhanced- and super-resolve data acquired from different fluorescence applications, we evaluated its performance over a collection of experimental scenarios (Figure 9) (Supplementary Note 10).
Analysis with MSSR provided nanoscopic resolution of rotavirus replication machineries (Figure 9a, Supplementary Figure S32), which were recently described by Garcés et al as a layered array of viral protein distributions [54]. Originally, it took the authors several days to weeks to generate a single extended-resolution image by means of analyzing several stacks of hundreds of DL images using 3B-ODE FF-SRM. With MSSR, we were able to achieve comparable results, through analyzing single DL frames within seconds with a regular desktop computer with either sf-MSSR1 or t-MSSR1 (Supplementary Note 7).
Mouse sperm cells are used to study the acrosomal exocytosis (AE), a unique secretory process which results from fusion events between the plasma membrane and a specialized vesicle called acrosome located in the sperm head [55, 56]. Nanoscopic remodeling of both plasma membrane and actin cytoskeleton was imaged during the AE by means of sf-MSSR1, showing single frame temporal resolution (of milliseconds) (Figure 8b, Supplementary Figures S33-34). At the onset of the AE, the FM4-64 fluorescence (a probe that fluoresces when bound to membranes) was confined to the plasma membrane and was visible above a F-actin cytoskeleton fringe. During the AE, several fenestration events were observed to occur at both the plasma and acrosome membranes, as consequence of that, a notorious increase of FM4-64 was observed close-bellow the F-actin fringe (Supplementary Movie S5 a-f). The AE is a dynamic remodeling process that takes minutes to occur, sf-MSSR1 allows the observation of events occurring at the millisecond scales, which are hindered when using other SRM multi-frame analytical approaches, such as SRRF or 3B [11, 57], due to their mandatory need of a temporal analysis of the fluorescence dynamics to unveil nanoscopic detail (compare Supplementary Figures S33 and S34).
Background noise is known to be an important issue in single-particle tracking (SPT) applications as it decreases the ability to faithfully localize particles and follow them through time [58, 59]. Moreover, the spatial overlap of PSFs derived from individual particles makes it challenging for SPT algorithms to recognize them as separate entities. The denoising capabilities of sf-MSSR1 enhanced both the contrast and spatial resolution of freely diffusing in-silico particles (Figure 8c, Supplementary Figure S35), previously used as benchmarks to test a variety of SPT algorithms [60].
Pre-processing of the images with sf-MSSR1 improved the tracking performance of three commonly employed SPT tracking algorithms: (i) the LAP framework for Brownian motion as in [61, 62], (ii) a linear motion tracker based on Kalman filter [63, 64], and (iii) a tracker based on Nearest neighbors [65–68] within a wide range of particle densities and SNR (Supplementary Figure S36). Additional testing with sf-MSSRn showed an increase in contrast for moving comet-like particles related to microtubule growth dynamics in live LLC-PK1 cells (Supplementary Figure S37 and Supplementary Movies S10 and S11), as well as higher nanoscopic colocalization accuracy in double imaging experiments in single-molecule DNA curtain assays (Supplementary Figure S38) [69].
Plasmalemma- and nuclear-labeled transgenic Arabidopsis thaliana plants are routinely used to study cell fate and proliferation during root development in time-lapse confocal microscopy experiments in two and three dimensions [70, 71]. When applied to lateral root primordium cells, located deep inside the parent root, sf-MSSR1 demonstrated the capacity to achieve multidimensional nanoscopic resolution as it revealed isolated nanodomains resembling nucleosome clutches, previously reported in mammalian cells [72, 73], within the nuclei of a lateral root primordium cells (Figure 8d-e, Supplementary Figure S39 and Supplementary Movie S12). Similar observations were performed upon epidermal root tissues visualized via selective plane illumination microscopy (SPIM) after examination of volumetric data with sf-MSSR1 (Supplementary Figure S40).
Supplementary Figure S41 shows a comparison of 3D reconstruction from a stack of epifluorescence images of bovine pulmonary artery endothelial (BPAE) cells. When a 2D image is processed by sf-MSSR1, an improvement in spatial resolution and contrast is observed (Supplementary Figure S41a,b). On the contrary, when a stack is used for a 3D representation (images taken at z-planes), the resolution obtained by sf-MSSR along the z-axis is not as refined as in the xy-plane (Supplementary Figure S41c,d). Nonetheless, a noticeable increase in contrast is attained in the overall dataset (Supplementary Figure S41, see Supplementary Movies S13 and S14). To further extend the resolution of DL images in 3D, it is necessary to extend the current implementations of the MSSR algorithm to account 3D information for MSSR processing.
In combination, these studies provide evidence for the capabilities of MSSR to resolve biological detail at nanoscopic scales using either simple or advanced fluorescence microscopy technologies.
Discussion
Novel theoretical contributions
From the historical point of view, since the seminal development of the MS theory [17, 18] and until the present day, few statistical and imaging applications based on the theory of MS compute the MS vector itself [74]. This can be explained, in part, because previous applications of MS are based on finding modes in the features space but did not calculate the MS vector, which is the key component of the working principle of MSSR. In contrast, MSSR represents an application of MS theory which also operates in the second derivative space. By computing the MS vector and estimating (photon) densities among pixels, MSSR computes a probability function for the fluorophore estimates whose individual fluorescence distributions are narrowed in comparison with the PSF of the optical system. The exploration of the information stored on the second derivative space of the image can be also achieved by substituting the MS by similar functions that operate in such space, e.g., Laplacian, Hessian, Difference of Gaussians [75] which, in comparison with the MS, offer computational advantages as they can be expressed in the Fourier space and implemented using the FFT algorithm [75]. The information harbored in the second derivative space of the DL image is used by MSSR to compute an image with higher spatial frequencies than the corresponding DL image, hence, overcoming both the Rayleigh and Sparrow limits, and setting up an undescribed limit of spatial resolution which deserves further exploration and characterization.
The MS theory is not restricted by the number of dimensions of the information required to compute the kernel windows over which MSSR operates (Supplementary Notes 2 and 3). Given that, MSSR parameters are suitable to extend its application to assess data with higher dimensions. For example, in 2D images, the spatial parameter of MSSR, which encompasses the lateral resolution width of the PSF, is defined to be the same in the x and y dimensions of the image. In such a case, the shape of the kernel is circle- or square-like, depending on the application used. For three-dimensional (3D) microscopy imaging, the lateral (x-y plane) and axial (x-z and y-z planes) dimensions are affected in different ways by diffraction. The MSSR principle can be further extended for explicit volumetric imaging by means of using an asymmetric kernel which can be defined following the 3D lateral-axial aspect ratio of the PSF (Supplementary Note 11). In addition, the definition of the spatial kernel can be refined to also consider possible deformations of axial symmetry of the PSF due to optical aberrations introduced by the imaging system or by the sample itself. A similar reasoning aimed to extend the portfolio of applications of MSSR can be envisaged considering spatial-range parameters, the latter narrowing down the working intensity space where local calculations of MSSR take place.
Novel contributions to microscopy
We present a new SRM approach capable of achieving multidimensional nanoscopy through single-frame analysis under low SNR conditions and with minimal noise-dependent artifacts. Limited only by the imaging speed of the optical system setup, MSSR increases resolution by analyzing either a single frame, or by applying MSSR to each individual image in a stack followed by the application of a pixel-wise temporal function. MSSR is a powerful stand-alone method for either single or multi-frame SRM approaches, or as a post-processing method which can be applied to other analytical multi-frame (restricted to camera-based systems) or hardware dependent SRM methods for further enhancement of resolution and contrast. We demonstrated MSSR compatibility with other SRM methods and showed that its usage enhanced resolution and overall image quality in all the cases tested.
FF-SRM analytical multi-frame approaches such as SRRF, ESI, MUSICAL and 3B demand a temporal analysis which limits their utility for multi-dimensional imaging of live samples [65]. The need to collect hundreds to thousands of images of the same pseudo-static scene, challenges the applicability of these methods in multidimensional imaging. The temporal multi-frame requirement imposes a tradeoff between the achievable temporal and spatial resolutions. MSSR removes these constraints while maintaining computational efficiency (Supplementary Note 7).
We present applications of the MSSR principle that revealed fast molecular dynamics through single-frame analysis of live-cell imaging data, with reduced processing times in comparison with similar SRM approaches (Supplementary Notes 7 and 10). Moreover, MSSR greatly improves the tracking efficacy of SPT methods by means of reducing background noise and increasing both the contrast and SNR of noisy SPT movies, enhancing the ability to resolve the position of single emitters. MSSR further pushes the limits of live-cell nanoscopy by its excellent single-frame performance. This flexibility extends its utility to most fluorescence microscopy and alternative SRM methods.
Achieving both high (or sufficient) temporal and spatial resolution within a broad range of fluorescence microscopy applications is a common goal among the bioimaging community. With recent advances in microscopy equipment and imaging protocols, the gap between the highest attainable resolution in the temporal and spatial dimensions within the same experiment, has narrowed. This has been a challenge especially because both parameters often involve mutually exclusive optical instrumentation and experimental strategies. The introduction of MSSR represents one more step in the right direction as it drastically reduces the amount of data needed to reconstruct a single super-resolved micrography.
No longer having to sacrifice either temporal or spatial resolution over the other, has led some scientists to propose new ways to analyze imaging data. Some approximations have been tailored to study millisecond molecular dynamics and structural feature changes within the same experiment [76], e.g., by taking advantage of the simultaneous use of image correlation spectroscopy (ICS) and FF-SRM methods such as SRRF [11]. In these contexts, MSSR could improve the analysis in three ways: it delivers reliable SRM images in low SNR scenarios, which are common in the experimental regimes of ICS due to the relatively fast frame rates of its applications, b) MSSR introduces no noise-dependent artifacts which further refines the quality of the spatial analysis and c) since no temporal binning is necessary for MSSR, there is no restriction in the level of temporal detail retrievable from the ICS analysis.
Sub-millisecond time-lapse microscopy imaging can now be achieved by sCMOS technologies, with applications for particle velocimetry [77], rheometry [78], and optical patch clamp [79]. We envisage further applications for MSSR in these areas through unveiling nanoscopic detail hidden in single DL images. Moreover, MSSR can facilitate correlative nanoscopic imaging through crosstalk with other imaging techniques such as electron microscopy, i.e., CLEM: correlative light electron microscopy [80]; or atomic force microscopy, i.e., CLAFEM: Correlative light atomic force electron microscopy [81]. In addition, MSSR can be applied to nanoscopic volumetric imaging by using it together with expansion microscopy [82], oblique angle microscopy [83], SPIM and lattice light sheet microscopy [84], extending their capabilities to previous unattainable resolution regimes.
A recent study by Chen R. et al., suggests that deep-learning based artificial intelligence (AI) can reconstruct a super-resolution image from a single DL image [85]. Such AI-based SRM approaches are promising, however, they are limited to the existence of a maximum likelihood image obtained with another SRM, such as STORM, that is required for neural network training and error minimization. Otherwise, the method is prompted to bias the final reconstruction toward the topological information used to train the AI - network [85]. Our approach works completely independent of other SRM methods and provides evidence of the existence of a new resolution limit which lies on the second derivative space of the DL image, information inaccessible when using neural networks.
MSSR applications might impact far beyond the field of microscopy, as its principles can be applied to any lens-based system such as astronomy [86] and high-resolution satellite imagery [87].
Author contributions
Conceptualization: ETG, AG.
Methodology: ETG, AG, RPC, VA, EBA, HTM, DT, AHC, RRM, JGD, RDA.
Investigation: ETG, AG, RPC, AHC, DM, VA, EBA, CCC, GVG, AHG, DT, JMR, YG, MGB, RRM, MJ, HTM, HOH, JOO, RDA, AB.
Visualization: ETG, AG, RPC, HTM, VA, EBA, AL, DT.
Supervision: AG, ETG, AHC, DK.
Funding Acquisition: AG, AD, MGB, JGD, CDW. Project Administration: AG.
Drafting Main Document: ETG, AG, AL, AHC, CDW.
Drafting Supplementary Material: ETG, AG, AL, RPC, RRM, JMR, DK, VA, EBA, HOH.
All authors contributed to the writing and reviewing of the manuscript.
Competing interests
The authors declare no competing interests.
Additional information
The online version contains supplementary methods, supplementary notes, supplementary movies, the MSSR manual for FIJI/ImageJ, MSSR scripts R, python MATLAB, and additional references.
Data availability
All raw imaging data which support the findings of this study are available from the corresponding author upon request. Source data are provided with this paper.
Correspondence and requests for materials should be addressed to A.G.
Code availability
Source code for R, python and MATLAB platforms is available as supplementary materials, the MSSR plugin for FIJI/ImageJ is available at https://github.com/MSSRSupport/MSSR.
Funding
This research was supported by Dirección General de Asuntos del Personal Académico (DGAPA) – Programa de Apoyo a Proyectos de Investigación e Innovación Tecnológica-UNAM (PAPIIT). and by Mexican Consejo Nacional de Ciencia y Tecnología (CONACyT) grant A1-S-9236 to JGD, grant number IN211821 to AG, IN211216 to C.D.W., IN204221 to JGD. Microscopy equipment was provided and maintained through CONACYT grants 123007, 232708, 260541, 280487, 293624 and 294781. CW and AG acknowledge the Chan Zuckerberg Initiative, grants:
GBI-0000000093 (to CDW and AG) 2021-240504 (to MGB and AG). MGB thank to
IBYME-CONICET: Grants: PICT 2017-3047 (Agencia Nacional de Promoción de la Investigación, el Desarrollo Tecnológico y la Innovación), Fundación Williams; Fundación Rene Barón. DK acknowledges the support of the National Science Foundation grant 2102832. AL acknowledges the Howard Hughes Medical Institute (HHMI Award GT12050-2019). We also acknowledge the Programa de Becas de Posgrado of CONACYT for granting scholarships to ETG, RPC, AL, DM, VA, EBA, CCC, GVG, DT, HTM, JLM, HH, and to JPO. YG acknowledges DGAPA/UNAM for postdoctoral fellowship.
Acknowledgements
To the Microscopy Facility at EMBL Rome and IBYME-CONICET for providing samples and materials for experiments. To the ECE and ICIMAF institutes for theoretical and experimental input to this work. The authors thank the LNMA staff for providing support with microscopy imaging. The OpenSPIM project was technically executed and extended by Oliver Valdez Escalona. We thank the Analytical and Quantitative Light Microscopy course staff from the Marine Biological Laboratory for providing samples and data collection assistance. All authors thank Paul Hernández, Luis Mochan, Federico Lecumberry, Angelica Flores Navarrete and Yuriney Abonza Amaro for their valuable feedback during the elaboration of the manuscript and review process. The comments raised by both reviewers of the manuscript increased the overall scientific quality of the manuscript. Thank you all for making it possible.
Footnotes
The changes to the manuscript are the following: Main manuscript New figures: - Figure 3: sf-MSSRn extends spatial resolution in confocal microscopy, - Figure 4: sf-MSSRn enhances the resolution and contrast of Airy scan and SIM reconstructions. - Figure 6: sf-MSSRn enhances spatial resolution in STED microscopy. - Figure 9: Showcase of MSSR wide-range fluorescence microscopy applications. Other figures have been updated. Supplementary material A description of Fourier interpolation and its comparison with bicubic interpolation (see section 6.2). otencial adaptation of MSSR to 3D images has been added to Supplementary (see section 11). New supplementary figures: - Supplementary Figure S1: Behavior of ks according to the distance in a neighborhood. - Supplementary Figure S8: Selection of spatial parameter hs related to FWHM. - Supplementary Figure S13: Thresholding and normalizing processes are not enough to overcome the Sparrow limit. - Supplementary Figure S14: Resolution increase provided by deconvolution methods, radiality maps and sf-MSSRn. - Supplementary Figure S23: Fourier interpolation in 2D. - Supplementary Figure S24: sf-MSSR processing times for bicubic and Fourier interpolations. - Supplementary Figure S25: Comparison between Fourier and bicubic interpolation for temporal analysis on experimental dataset. - Supplementary Figure S37: Nanoscopic, single-frame, live-cell imaging of microtubule dynamics in LLC-PK1 cells. - Supplementary Figure S41: Comparison DL images of sf-MSSR reconstruction of epifluorescence BPAE cells for 2D and 3D images. - Supplementary Figure S44: Temporal fluctuation characteristics of two GattaPaint Nanorulers. Most of the figures have been updated with the mpl-infierno LUT using FIJI/ImageJ. Manual Figures 3, 5, 6, 8 and 9 have been updated. Movies New movies: - Supplementary Movie 10: sf-MSSR video of live LLC-PK1 cells expressing mEmerald-EB3. - Supplementary Movie 11: sf-MSSR video of an apoptotic LLC-PK1 cell expressing mEmerald-EB3. - Supplementary Movie 13: DL reconstruction of epifluorescence BPAE cells for 2D and 3D images. - Supplementary Movie 14: sf-MSSR1 reconstruction of epifluorescence BPAE cells for 2D and 3D images. - Other movies have been updated. MSSR Plugin Fiji/ImageJ and other codes With the new changes MSSR Plugin has been updated to version 2.0.0. "PSFp" parameter has been redefined as "FWHM of PSF". An option to select the type of interpolation has been added. An option related to perform intensity normalization on sequence of images has been added. Codes for Matlab, R, and Python have been updated according to the new changes on the MSSR theory.
References
- 1.↵
- 2.↵
- 3.↵
- 4.
- 5.↵
- 6.↵
- 7.
- 8.
- 9.↵
- 10.↵
- 11.↵
- 12.
- 13.↵
- 14.↵
- 15.↵
- 16.↵
- 17.↵
- 18.↵
- 19.↵
- 20.
- 21.↵
- 22.↵
- 23.
- 24.↵
- 25.↵
- 26.↵
- 27.
- 28.↵
- 29.↵
- 30.↵
- 31.↵
- 32.↵
- 33.↵
- 34.↵
- 35.↵
- 36.↵
- 37.↵
- 38.↵
- 39.↵
- 40.↵
- 41.↵
- 42.↵
- 43.↵
- 44.↵
- 45.↵
- 46.↵
- 47.↵
- 48.↵
- 49.↵
- 50.↵
- 51.↵
- 52.↵
- 53.↵
- 54.↵
- 55.↵
- 56.↵
- 57.↵
- 58.↵
- 59.↵
- 60.↵
- 61.↵
- 62.↵
- 63.↵
- 64.↵
- 65.↵
- 66.
- 67.
- 68.↵
- 69.↵
- 70.↵
- 71.↵
- 72.↵
- 73.↵
- 74.↵
- 75.↵
- 76.↵
- 77.↵
- 78.↵
- 79.↵
- 80.↵
- 81.↵
- 82.↵
- 83.↵
- 84.↵
- 85.↵
- 86.↵
- 87.↵