Deep learning-based aberration compensation improves contrast and resolution in fluorescence microscopy

Optical aberrations hinder fluorescence microscopy of thick samples, reducing image signal, contrast, and resolution. Here we introduce a deep learning-based strategy for aberration compensation, improving image quality without slowing image acquisition, applying additional dose, or introducing more optics into the imaging path. Our method (i) introduces synthetic aberrations to images acquired on the shallow side of image stacks, making them resemble those acquired deeper into the volume and (ii) trains neural networks to reverse the effect of these aberrations. We use simulations to show that applying the trained ‘de-aberration’ networks outperforms alternative methods, and subsequently apply the networks to diverse datasets captured with confocal, light-sheet, multi-photon, and super-resolution microscopy. In all cases, the improved quality of the restored data facilitates qualitative image inspection and improves downstream image quantitation, including orientational analysis of blood vessels in mouse tissue and improved membrane and nuclear segmentation in C. elegans embryos.

Omitting the (, , ) coordinates for simplicity, the forward imaging model can be Fourier transformed (ℱ( )) to obtain where the Fourier transform of the PSF, ℱ() is also known as the optical transfer function (OTF).If () = 0, i.e., the wavefront is 'flat' or unaberrated, we obtain the diffraction-limited image  0 from the ideal PSF  0 and object  as Eliminating the object term using the last two equations, To prevent division by zero, we modify the denominator by adding a small value  ( = 0.01 for all datasets in this paper).Inverse Fourier transforming and optionally adding a noise term we obtain finally This equation provides a prescription for deriving an aberrated image  from a diffraction-limited image  0 and the ratio of aberrated OTF ℱ() to diffraction limited OTF ℱ( 0 ).The degree of aberration can be tuned by adjusting the Zernike functions   () and magnitude of their associated coefficients   .
In practice, we use images from the near side of fluorescence microscopy volumes as  0 and tune   until we obtain  that resemble aberrated images that occur on the 'far side' of the image stack (see Supplementary Table 1 for relevant parameters used in this paper).

Extension to different microscopes
To adopt this concept for other microscopes, we modify the PSF in equation ( 8), extending the wide-field PSF as needed to more accurately model light-sheet microscopy, confocal microscopy, instant SIM, and two-photon microscopy.Correspondingly, we re-write the equation ( 8) by adding a subscript "WF" to explicitly indicate it as using the wide-field PSF: We leave (, , ) as the generalized system PSF, which is constructed by considering the excitation PSF and emission PSF: By substituting equation ( 16) into equation ( 10), we obtain the forward model of each microscope.

1) Light sheet microscopy
In light sheet microscopy, the emission PSF is equivalent to the wide-field PSF and the excitation PSF is often (as in these experiments) modeled as a virtual sheet constructed by scanning a low numerical aperture Gaussian beam across the field of view.We model the excitation sheet as uniform in the lateral directions but Gaussian in the axial direction (lateral and axial are defined from the perspective of the detection objective): Then the final system PSF is where  is a constant which is discarded as we normalize the system PSF by integrating its intensity to 1.
In practice, by measuring the thickness of the light sheet, i.e., the full width at half maximum (FWHM) in the axial direction at the beam waist,  can be estimated based on the assumption that the beam is Gaussian as:

2) Confocal microscopy
We construct the confocal PSF as: where  (1) and  (2) are wide-field PSFs with excitation wavelength  1 and emission wavelength  2 , respectively and ⨂ is the convolution function. ℎ models the physical pinhole in a confocal system as a binary circular mask at the z=0 plane: In practice, we implement the convolution operation in equation ( 20) in the Fourier domain, giving: Note that we do not model the multifocal and pinhole lattice in spinning-disk confocal microscopy, so the model used for this form of microscopy is the same as what is used in point-scanning confocal microscopy.

4) Two-photon microscopy
For two-photon microscopy, the excitation PSF is the square of the wide-field PSF with the excitation wavelength.The emission PSF is treated as uniform as there is typically no confinement or modulation on the emission side, and this constant can be discarded as the final system PSF is normalized as described above.Therefore, we have: For most datasets employing single photon microscopy, we used 488 nm excitation and 532 nm excitation; for two-photon microscopy data, we used 960 nm wavelength excitation.

Supplementary Fig. 1 ,
DeAbe performance depends on aberration magnitude in input images and training data.A) Example aberrated PSFs and associated aberrated input images (left); DeAbe model predictions yielding aberration corrected images (middle), shown for different training data consisting of mixed aberrations with indicated maximum root mean square (RMS) wavefront distortion (see also Methods); and ground truth synthetic objects (right).The example input aberration here is defocus, applied at an increasing aberration magnitude (0.5 rad -4 rad RMS wavefront distortion).b) Quantification of a).Structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) metrics are used to quantify the quality of DeAbe Net predictions under different training data regimes (green, blue, red curves at 1, 2, and 4 rad max RMS) vs. ground truth.All DeAbe predictions show improvement compared to input aberrated image (black), with training data containing larger magnitude aberrations better able to compensate for larger aberration magnitudes in the input images.For this work, we used the model corresponding to a maximum RMS wavefront distortion of two radians (blue curve).Means and standard deviations are shown from 100 independent simulations, each with randomized object structures and randomized input aberrations.Scale bar: 5 µm.Supplementary Fig. 2, Dependence of DeAbe prediction on number of Zernike orders in training data.Synthetic aberrated phantom structures in Fig. 1b were input into DeAbe networks trained with progressively more Zernike basis aberrations.Visual a) and quantitative b) analysis indicate that performance plateaus after the 4 th Zernike terms are added.Thus, in this work, we used four Zernike orders to generate training data.In b), both SSIM (left) and PSNR (right) values are shown from 100 independent simulations.Scale bars: 5 µm and 2.5 µm (insets).Supplementary Fig. 3, DeAbe network predictions on images contaminated with specific aberration modes.Synthetic phantoms were aberrated with defocus, coma, spherical, astigmatism, trefoil, and mixed aberrations with indicated RMS wavefront distortion.A DeAbe model trained on data contaminated with mixed aberrations (up to fourth order Zernike basis terms, maximum RMS wavefront distortion two radians) was then used to compensate for aberrations.a) Aberrated PSFs (left), aberrated images (middle) and DeAbe model predictions (right) are shown for each condition, as well as the ground truth reference (GT).Z in each vertical axis label refers to the Zernike order.b) Quantification using SSIM and PSNR metrics (means and standard deviations from 100 independent simulations) support the visual improvement after application of the DeAbe model.Scale bar: 5 µm.Supplementary Fig. 4, DeAbe network predictions on defocused images improve if using a dedicated network trained purely on defocused images.Synthetic phantoms were aberrated with defocus with indicated RMS wavefront distortion.A general DeAbe model trained on data contaminated with mixed aberrations (up to fourth order Zernike basis terms, maximum RMS wavefront distortion two radians), or a specific defocus DeAbe model trained on data contaminated with only defocus aberrations (maximum RMS wavefront distortion two radians), was then used to compensate for aberrations.a) From left to right: aberrated PSFs, aberrated images, predictions with a general DeAbe model, predictions with a specific defocus DeAbe model, and the ground truth reference (GT).b) Quantification using SSIM and PSNR metrics (means and standard deviations from 100 independent simulations) to compare the performance of the general DeAbe model vs. the specific defocus DeAbe model.Scale bar: 5 µm.Supplementary Fig. 8, Progressive improvements in image quality in NK-92 cells imaged with iSIM.Cells were fixed and stained for wheat germ agglutinin as in Fig. 2b-d.Columns show (left to right) representative images of raw data, deconvolved data, DeAbe prediction on raw data, and DeAbe prediction followed by deconvolution (20 iterations Richardson-Lucy).The latter provides the clearest lateral (a-c) and axial (d) views, best highlighting fine features in the data (arrows).MIP: maximum intensity projection.Scale bars: 5 µm.See also Supplementary Video 5. simulation of oriented lines with more (left) or less (right) mean directional variance (DV).More directional variance implies lines are more randomly oriented.c-e) Orientation analysis on blood vessel channel of DeAbe processed data of CLARITY-cleared E11.5-day mouse embryo, immunostained for neurons (TuJ1) and blood vessels (CD31).Perspective views of  (c),  (d) and DV (e) distributions in 3D are shown.f) Histograms of  (top left),  (top right), and DV (bottom).See also Fig. 3e-g, Supplementary Video 9. Scale bars: 500 µm.Supplementary Fig. 13, Multi-step image restoration schemata.We trained independent neural networks to compensate for aberrations (Step 1: DL DeAbe, see also Fig. 1a), deconvolve the data (Step 2: DL Decon), and to super-resolve the data (Step 3: DL Iso, a) or DL Expan, b)).For the deconvolution networks, we used high quality joint deconvolution based on orthogonal views (after DL DeAbe processing) from diSPIM as ground truth and trained the network to predict this joint deconvolution given only single-view input.For the isotropic enhancement model in a), we synthetically blur lateral views to resemble axial views and downsample and upsample the data to mimic the coarser pixel size in z.We then train a neural network to reverse this degradation.For the expansion microscopy network in b), we synthetically degraded high-resolution images from expanded samples until the synthetic data resembled conventional images acquired on the diSPIM and trained a network to reverse this degradation.Serial application of each network produced the final prediction.See also Methods.(columns).Merged views (top), membrane channel (middle), and nuclear channel (bottom) are shown.Scale bar: 5 µm.Supplementary Fig. 16, Automated membrane segmentations on restored image volumes derived from dual-color (pan-nuclear and membrane) C. elegans embryos.a, b) Image planes from nuclear and membrane channels from representative volume as in Fig. 4a-d, after multistep image restoration.c) Membrane-enhanced image after applying vascular structure enhancement filter to b. d) Segmented nuclei with Mask RCNN.e) Overlay of enhanced membrane and segmented nuclei.f) The associated multiinstance segmentation of cell boundaries.Scale bar: 5 µm.Supplementary Fig. 17, Ablation experiments highlighting value of DeAbe model in multi-step restoration.Top rows show data from membrane channel of C. elegans embryos as in Fig.4a, highlighting progressive improvements in image quality after each step of multi-step restoration, including higher magnification insets in second row corresponding to dashed rectangular regions in top row.Data are also shown (i) if DeAbe (Step 1) is ablated, proceeding directly from raw data to Steps 2 and 3; (ii) if DeAbe is ablated, and joint deconvolution used in lieu of the DL Decon (Step 2); (iii) if steps 1 and 2 are both ablated, and raw data is used directly as in the input for the DL Iso network (Step 3).The final prediction is degraded in schemes (i) -(iii) relative to the original method, as membranes are either noisy or not well resolved (arrows).Scale bars: 5 µm, 2 µm inset.Supplementary Fig.18, Using expanded embryos as high resolution ground truth for training resolution enhancement model.a) Expansion was used to enlarge embryos expressing ttx-3B-GFP 3.6fold and the samples were then imaged using diSPIM.The two views (A and B) were registered, and joint deconvolution applied to produce volumes with more nearly isotropic resolution.These volumes were synthetically degraded via blurring and downsampling to produce semisynthetic data with resolution more like the raw data prior to expansion, and used to train a network (Expan model) for resolution enhancement.In these images, GFP signal was boosted using immunolabeling.b) Example images from two embryos highlighting nerve ring region (orange dashed rectangles), comparing synthetic, lowresolution input data (left), prediction from DL Expan model, and ground truth data.The Fourier transform images are shown in the insets, with cyan, pink, and black dotted circles corresponding to the spatial resolutions of 330 nm, 130 nm, and 105 nm respectively.Scale bars: 2 µm.right, comparing raw (upper) and restored (lower) data.AIY and SMDD cell bodies have been indicated with arrows for clarity; dashed circles highlight associated neurites as they innervate the nerve ring area.Scale bars: 5 µm.Supplementary Fig. 22, Multi-step image restoration does not suffer motion artifacts like multiview fusion.a) Maximum intensity projection (MIP) images of C. elegans embryos expressing ttx-3B-GFP, imaged with diSPIM, comparing raw single-view recordings (Raw, left), dual-view fusion result via traditional joint deconvolution (Joint Decon, middle) and deep learning-based deconvolution after application of DeAbe (DL Decon, right).Higher magnification lateral views of orange dotted rectangular region are shown in the bottom panel, with cyan and red arrows highlighting motion induced artifacts present in Joint Decon images but not Raw or DL Decon images.b) Images of C. elegans embryos expressing pan-nuclear GFP marker, imaged with symmetric diSPIM.Higher magnification lateral views of green dotted rectangular region are shown in the bottom panel, with red dotted circles highlighting the motion induced artifacts present in Joint Decon images but not Raw or DL Decon images.Scale bars: 5 µm, 2 µm insets.Supplementary Fig. 23, Additional example of multi-step restoration.a) C. elegans embryos expressing pan-nuclear GFP marker were imaged with high numerical aperture diSPIM and input into our multi-step imaging restoration pipeline.Top row: Comparing raw, after DeAbe (Step 1), after additionally applying deconvolution network (Step 2), and finally after additionally applying DL Expan model (Step 3).See also Methods for further details on sample preparation and imaging system.Higher magnification views of lateral (left, corresponding to dashed orange rectangle in top row) and axial (right, corresponding to dashed blue rectangle in top row) maximum intensity projections (MIP) are also shown.b) As in a), but highlighting single lateral and axial imaging planes instead of MIPs.Scale bars: 5 µm, 2 µm insets.See also Supplementary Videos 13, 14.

Supplementary Table 1, Sample information and parameters used in generating DeAbe models.
All samples and datasets used for training the DeAbe models in this paper.The time costs are reported based on tests on a Windows 10 workstation (CPU: Intel Xeon, Platinum 8369B, two processors; RAM: 256 GB; GPU: NVIDIA GeForce RTX 3090 with 24 GB memory).With additional footnotes: 1: The training data consists of paired GT images and synthetically aberrated images.2: Representative single volume for testing the applying time cost.3: For multicolor images, data size and time cost are reported for single color channel.