Abstract
In fluorescence microscopy, the amount of information that can be collected from the sample is limited, often due to constraints imposed by photobleaching and phototoxicity. Here, we report an event-driven acquisition (EDA) framework, which combines real-time, neural network-based recognition of events of interest with automated control of the imaging parameters in an instant structured illumination microscope (iSIM). On-the-fly prioritization of imaging rate or experiment duration is achieved by switching between a slow imaging rate to detect the onset of biological events of interest and a fast imaging rate to enable high information content during their progression. In this way, EDA allows the data capture of mitochondrial and bacterial divisions at imaging rates that match their dynamic timescales, while extending the accessible imaging duration, and thereby increases the density of relevant information in the acquired data.
Live-cell fluorescence microscopy is an indispensable tool for studying the dynamics of biological systems. Individual molecules can be labelled with fluorescent tags to allow high-lighting of specific molecular species. Yet, these approaches come with important limitations: as fluorophores go through multiple excitation and emission cycles, they risk photobleaching and producing reactive oxygen species, which interact with cellular components, causing phototoxicity. These constraints set a finite photon budget, limiting the information that can be gathered from the sample, and confining the accessible imaging parameter space. Practically, this is manifested by a trade-off between spatial resolution and imaging rate, duration and the signal-to-noise ratio (SNR), such that improving one parameter comes at a cost to the others (1, 2). In response, smart and adaptive microscopies have been developed to navigate the parameter space more efficiently and improve imaging performance in changing biological samples. Adaptive light sheet microscopy has been used to compensate for changes in the optical properties of a developing organism, and optimize signal resolution on-the-fly (3). Other approaches have controlled where or how light is delivered to the sample in response to the local signal, to illuminate where the labelled structures reside and reduce the overall light dose (4), while improving the dynamic range (5) or imaging speed (6, 7).
While the spatial resolution and SNR are largely imposed by the method of choice (8), increasing the imaging repetition rate causes sample health and photon budget to degrade faster, thus reducing the imaging duration. Given these dependencies, high frame rates and long-term fluorescence imaging are mutually incompatible, and microscopy users are obliged to compromise to capture events of interest under conditions that preserve sample health. Biological processes, however, are not bound by such technical limitations and take place across a range of temporal scales with varying dynamics. This is exemplified by rare but rapid events such as mitochondrial division, or events with a low duty cycle such as the final stages of constriction within the bacterial cell cycle. Capturing such events is challenging, as the optimal frame rate may change during a single acquisition depending on the state of the sample and the information about events of interest contained within. Slow frame rates can better probe for events of interest over longer times while preserving sample health; while faster imaging better captures the short-lived dynamic intermediates, enriched in the event-specific information content. However, current imaging methods acquire data at a fixed frame rate regardless of sample information content and hence risk missing events of interest either because of short imaging duration or insufficient temporal resolution. Despite advances in adaptive microscopy to optimize image quality and reduce photobleaching, there is currently no approach to adapt the acquisition parameters to the presence of events of interest.
We propose an event-driven acquisition (EDA) framework which links the distinct spatio-temporal signatures of biological events to the choice of microscope imaging parameters – as demonstrated here, by adapting the acquisition frame rate. In response to events of interest detected in real time by a neural network, EDA prioritizes the imaging speed or duration on-the-fly. This allows us to capture the dynamic behaviour of events of interest and collect more images with a higher information content, while extending the imaging duration and preserving sample health by slowing down the acquisition during a lack thereof. The framework therefore spends the sample’s photon budget more efficiently and increases the information that we are able to gather from the sample before reaching the photobleaching or phototoxicity threshold. As proof of principle, we integrate EDA into the controller for a large field-of-view (FOV) instant structured illumination microscope (iSIM) (9, 10) and apply it to capture super-resolved time-lapse movies enriched in mitochondrial and bacterial divisions.
The EDA framework integrates a feedback loop between the sample and microscope controls, the implementation of which requires: 1) sensing via imaging to collect information from the sample, 2) computation to detect events of interest and 3) actuation to adapt the imaging speed (Fig. 1a). For our setup, the acquisition is performed using MicroManager, which saves captured frames to a network attached storage unit. As new frames are added, the computation step is triggered on an external server, which generates spatial probability maps for a specific event of interest and sends back the maximal value, corresponding to the information content found in the corresponding image. Depending on this decision parameter, the microscope controls then generate the acquisition sequence for the next frame(s): a fast imaging speed is triggered if the event probability exceeds a set threshold, or once in the fast mode, the microscope switches back to the slow imaging speed if the readout goes below a second threshold, set slightly lower (Fig. 1b,c).
a Feedback control loop between the sample and the acquisition parameters composed of three main parts: 1) sensing by image capture to gather information from the sample 2) computation using a neural network to detect events of interest to generate a probability map and 3) adaptation of the acquisition parameters in response to the sample. b Schematic representation of the trade-off between the imaging speed and light exposure over the duration of an imaging experiment. The total amount of photon budget available (shaded areas) stays the same for all techniques. c Example of event probability (computation, black) as a function of time obtained during an EDA-guided iSIM imaging experiment, and the adaptive imaging speed (actuation, red). Frames of events of interest that triggered the change of the imaging speed and the corresponding probability maps are represented on the top (5.2 µm × 5.2 µm).
We applied EDA to measure mitochondrial constriction and division. Such events are challenging to capture, because they are infrequent and can occur at any location within the mitochondrial network distributed throughout the cell. Once initiated, constriction can proceed to division within tens of seconds (11). To detect events of interest, we trained a neural network with a U-net architecture (12) to recognise mitochondrial constrictions in the presence of dynamin related protein 1 (DRP1). Although DRP1 accumulation is required for most fissions (13), it also collects at non-dividing sites through its curvature-sensing properties. Thus, its presence alone is not predictive of a future division, making a simple thresholdingbased adaptive control ineffective. The network produces a probability map, marking potential constriction sites across the imaging FOV. The maximum value in this map represents the most pronounced constriction site, and hence most informative, which is then used as a deciding factor in the actuation step (Sup. Note 2).
Using the detection network, we performed EDA with the iSIM on Cos-7 cells expressing mitochondrion-targeted TagRFP and DRP1-Emerald. The network recognised several instances containing events of interest during imaging periods of up to 10 minutes (Fig. 2a). To compare the performance of EDA to traditional imaging acquisitions using fixed parameters, we also collected data at a fixed imaging speed of either 0.2 frames/sec or 4 frames/sec, representing slow and fast imaging speeds respectively. EDA showed considerably less photobleaching decay, with an average decrease in the exponential bleaching decay constant of 3.4-fold (Fig. 2b, c). While the bleaching decay is still on average 10 times higher compared to fixed slow imaging alone, it was possible to reach comparable 10-minute imaging durations for many of the EDA experiments. Importantly, during events of interest (Fig. 2c), we were also able to better capture the dynamic intermediates of membrane remodelling, reflected in the measured constriction diameters, which had significantly smaller values with EDA (Fig. 2f). This reflects the ability of EDA to better resolve the most constricted states preceding division and prioritize gathering data with the highest information content for our event of interest. Furthermore, considering the lower average photobleaching, we find that EDA uses the sample’s photon budget more efficiently by collecting more images with a higher information content while reducing the sample light exposure. Finally, to assess the differences in information throughput, we quantified the information content of the recorded images based on the output of the event detection network (Sup. Note 2). These results demonstrate that EDA produced data with significantly higher information content, compared to fixed slow or fast imaging alone (Fig. 2h).
a Representation of the different imaging speeds in an event-driven acquisition (Scale bar: 1 µm). Times of the individual frames are represented approximately on the timeline. b, d Decay constants obtained from fitting the mean signal in the mitochondria over time with an exponential decay. (See also Sup. Figure 1) c, e Cumulative light dose, defined as the summed laser power multiplied by the exposure time over frames, over the duration of an EDA or fixed frame rate experiment. EDA sometimes achieves a higher total light dose due to the recovery effects during slow imaging speed phases (See Sup. Figure 1). f, g Minimal constriction width measured when following a constriction event over time. h, i Information density, calculated as the amount of data generated per movie and weighed by the information throughput of each frame, measured for EDA and traditional imaging approaches using fixed low or high imaging speeds. (Mitochondria: slow: N = 360 frames in n = 3 independent experiments; fast: N = 765, n = 2; EDA: N = 1516, n = 4; C. crescentus: slow: N = 228, n = 2; fast: N = 296, n = 3; EDA: N = 182, n = 5).
To showcase the generalizability of EDA, we extended its application to the final stages of bacterial division by imaging Caulobacter crescentus cells expressing a cytoplasmic mScarletI, and FtsZ-sfGFP to mark the division site (Sup. Note 1). We found that our event detection network developed for mitochondrial constrictions could also recognise the final stage of bacterial division (Sup. Note 2), likely due to morphological similarities in constriction shape and the presence of a functionally similar molecular marker. We collected data on cells at a fixed slow imaging speed of 6.7 frames/hour, fast imaging speed of 20 frames/hour, or using EDA to switch between the two (Fig. 2a). The bleaching decay was improved over the fixed fast imaging by a factor of 1.7, with double the useful imaging duration (Fig. 2d, e). Constrictions could again be measured down to significantly smaller diameters (Fig. 2g) and the gain in information throughput was comparable to that estimated in the mitochondrial experiments (Fig. 2i). This confirms that EDA improves access to the details of bacterial cell division events that would be difficult to capture using a fixed imaging speed, and uses the photon budget more efficiently. We found that EDA was especially beneficial for imaging synchronized bacterial populations, where it sensed the onset of divisions and enabled high-contrast and high-frame rate imaging for many divisions simultaneously (Sup. Figure 2).
Overall, our results highlight the advantages of implementing the EDA framework for different biological systems operating at distinct timescales. Imaging of mitochondrial divisions benefits from long imaging durations to allow for such rare events to develop, and fast imaging speed once constrictions form to capture the structural intermediates of membrane remodelling. For Caulobacter crescentus, EDA allows us to follow the entire cell-cycle and image the final states of division at high temporal resolution. In both systems, as shown from the measured constriction diameters, EDA data gathered from individual cells contained smaller constriction sizes, difficult to capture by traditional imaging approaches due to their transience. Beyond that, the benefit is two-fold, since EDA also generates data with higher information density, and hence lighter and more useful datasets. The modular nature of our implementation of the framework will aid adaptation of EDA to different imaging methods and biological systems.
While our implementation adapts the imaging speed in a digital manner, EDA could evoke more complex responses such as multi-level or analog actuation, as well as controlling other acquisition parameters such as the excitation power or exposure time. Potential pitfalls include the need to decouple experimental variability from real biological changes. As phototoxicity is thought to vary non-linearly with the delivered light dose (14), individual acquisitions using EDA will differ in imaging conditions based on the information content of the sample, which should be kept in mind when comparing datasets. Furthermore, the benefits of applying EDA depends on the specific dynamics of the observed events – the greatest advantages are expected if it’s possible to choose a large difference in imaging speeds. Brief, dynamic and rare events that can be predicted early are therefore best suited for observation using EDA.
Our work builds on the developing field of ‘smart’ or ‘self-driving’ microscopes (15), designed to adapt the acquisition procedure on-the-fly to either preserve the sample from unnecessary illumination (4, 5, 7), generate faster (6, 16) or better quality data (3, 17). However, while these methods focus on adapting the imaging parameters for optimized image quality, EDA is unique in that it integrates the information about the biological state of the sample into the acquisition procedure. Nevertheless, the EDA framework is generalizable to other microscopes capable of on-the-fly image processing, and was developed particularly for applications with changing dynamics. Moreover, machine learning has proven useful for many image analysis tasks, allowing the detection of increasingly complex sample features. As event recognition networks become increasingly accessible, EDA can enhance the information content of gathered data by repurposing such networks at the acquisition step. With this, we expect that EDA will boost biological discovery by delivering light-weight, information-rich datasets of events that are otherwise challenging to record.
Methods
Sample preparation
Cos-7
African green monkey kidney (Cos-7) cells were prepared as described in (11).
Dual transfections using cox8-TagRFP and Emerald-Drp1 were conducted using Lipofectamine 2000 (Life Technologies). Transfections were performed in Opti-MEM using 150 ng of plasmid and 1.5 µl of Lipofectamine 2000 per 100 µl Opti-MEM. Imaging was performed at 37°C in pre-warmed Leibovitz medium on a 25 mm, #1.5 glass coverslip (Menzel).
Caulobacter crescentus
Liquid C. crescentus cultures were grown overnight at 30 °C with 3 mL of 2xPYE medium under mechanical agitation (200 rpm). Liquid cultures were re-inoculated into fresh 2xPYE medium to grow cells until log-phase (OD660=0.2-0.4). For antibiotic usage, 5 µg/mL kanamycin and 1 µg/mL gentamicin were added in liquid cultures. To induce the expression of FtsZ-sfGFP and mScar-letI under the Pxyl and Pvan promoter respectively, 0.5 mM of vanillate and 0.02% xylose were added to the culture 2 hours before imaging or synchronization (18).
C. crescentus cell cultures were spotted onto 2xPYE agarose pad for imaging. To make the agarose pad, a gasket (Invitrogen™, Secure-Seal™ Spacer, S24736) was placed on a rectangular glass slide, and filled with 1.5% 2xPYE agarose (Invitrogen™, UltraPure™ Agarose, 16500100) containing 0.5 mM of vanillate and 0.02% of xylose without adding any antibiotics. Another glass slide was placed on the top of silicone gasket, and the sandwich-like pad was put into fridge for agarose solidification. After 20 min, the top cover slide was removed, 1-2 µl drops of cell suspension was placed on the pad. After full absorption of the droplet, the pad was sealed with a plasma-cleaned #1.5 round coverslip of a diameter of 25 mm. Notably, for imaging the synchronized cells, 1xPYE agarose pad was used to let cells grow slower.
iSIM imaging
Imaging was performed on a custom built instant structured illumination microscope (iSIM) that is described in detail in (10). Emerald/sfGFP and TagRFP/mScarletI fluorescence was excited using 488 and 561 nm lasers respectively. A system of microlens and pinhole arrays together with a galvo actuated mirror allows for instant super-resolution imaging on the camera chip (Photometrics Prime95B sCMOS). The implementation of a mfFIFI setup allows for homogeneous excitation over the full field of view (10).
SIM imaging and training data
Training data was acquired at the fast SIM using a SLM and LCD for generating and controlling the illumination pattern (19). The training data consisted of constriction events published previously (11).
Event detection
Detection of potential division events was performed using an implementation of the U-net (12). The network was trained on 29600 dual-color images of mitochondria and Drp1, and frames marking the respective locations of divisions (ground truth) (Sup. Note 2). This data set was recorded on a fast dual-color SIM setup at Janelia Farm (19) and published previously (11). The ground truth was computed from the intensity in the Drp1 channel, multiplied by the negative product of principal curvatures of the mitochondrial signal (Sup. Note 2). This information was then curated manually to increase accuracy for real constriction events by visually removing false positives or ensuring consistent detection of active constriction sites. With this, the network reaches an accuracy of 87% when tested on data that was not used for training. The output of inference is a two dimensional map of division probabilities in the range 0 to 255. Training was performed using the tensorflow/keras module in python 3.9 (20).
Event-driven acquisition for adaptive temporal sampling
The different parts of the EDA framework were implemented in separated modules that allowed for continuous and independent testing of the components.
Data handling
The EDA framework is distributed over Micro-Manager (21) for general microscope handling, and Matlab (MathWorks) for the control of the timing of the microscope components and Python for network inference. Furthermore, the Python module was used on a machine in the network due to hardware restrictions on the local computer used for microscope control. A network attached storage (NAS) was used to allow for communication of the different components of the system over the local 10 Gbit network. Recorded frames were stored as single .tif files to the NAS by Micro-Manager. A server implemented using the watchdog module (Python) on the remote machine detected new files for inference. After calculation of the decision parameter, the value was saved to a binary file that was used by Matlab to calculate the values for the next sequence of imaging.
Event Server
When a new file for each channel respectively was detected on the NAS, the event server implemented in python running on a remote machine first performed data preparation on the recorded frames. The frames were resized by a factor of 0.7 to match the pixel size of the training data. Both frames were smoothed using a Gaussian of 1.5 px with an additional background subtraction in the Drp1 channel (Gaussian of 7.5 px). The frames were tiled into overlapping 128×128 px sized tiles to match the size of the training data. The individual tiles of the structure (mitochondria/Caulobacter) channel were again normalized individually. Inference was performed on pairs of structure/foci tiles and the output was stitched together. The maximum value from the stitched frame was recorded as the highest probability in the frame for a division event and saved to the binary file on the NAS.
Hardware control
The timing of the microscope hardware is performed using a PCI 6733 analog output device (National Instruments). The device is used in background mode requesting data when the existing sequence is ending. For mitochondria imaging, either a sequence containing one frame over five seconds is provided in the slow mode, or a sequence containing five frames in one second. For bacterial imaging, sequences with one frame over 3/9 or 2/12 minutes is provided for slow/fast and normal or synchronized colonies respectively. The parameters are chosen depending on which value was read from the binary file on the NAS that contains the latest event information written by the event server. In addition to a threshold, a hysteresis band is implemented here by defining an upper and a lower threshold. Fast acquisition starts at surpassing the upper threshold and is only stopped when the event probability falls below the lower threshold. For Caulobacter imaging, the number of fast frames was further set to a minimum of three.
Due to the fast imaging rates in the mitochondria imaging, the new sequence is calculated before the event server has calculated the probability map for the last frames, leading to a delay in the reaction of the EDA framework. This is overcome for the Caulobacter imaging by delaying the calculation of the new sequences by 10 seconds allowing for mode switching on the newest data available.
Data Analysis
Photobleaching Decay
The different photobleaching kinetics of the modes were characterized by the intensity contrast of the samples. The channel of the structural feature (mitochondria and caulobacter outline) was segmented using a otsuthresholding method after a median and Gaussian filter were applied (kernel size of 5 px each). The intensity contrast was then calculated as the mean intensity in the segmented region divided by the mean intensity in the rest of the image. The decay constant was then obtained by fitting an exponential decay function (y = a exp(−bx) +c) to the intensity contrast over time data and extracting the b term.
Cumulative Light Dose
The total amount of light exposure over time was calculated from the number of previously recorded frames. Traces were truncated after a minimal intensity contrast was reached (1.1 for mitochondria and 1.02 for Caulobacter).
Constriction Width
The time resolution of the slow and EDA modes was evaluated by calculating the minimal width of constriction measured during the experiment. The measurement was performed similar to the method described in (11). Parameters are the ones used for the mitochondria analysis with Caulobacter values in brackets. The deconvolution of the images was performed using the Richardson Lucy algorithm as implemented by the flowdec python package (22). Events of interest were defined by a minimum value of the neural network output of 80 (90 for C. crescentus). A number of subsequent frames at the same position were analyzed until a maximal observation time of 20 seconds (1 hour for C. crescentus) was reached. Segmentation, skeletonization and spline fitting the resulting points lead to a backbone of the mitochondrion in a frame of 20 by 20 pixels around the position of the detected event. 100 perpendicular lines were calculated around the closest point of the backbone to the event position. The intensity profile along those lines was fitted using a gaussian profile. The full width at half maximum (FWHM) of the gaussian profile with the smallest σ was recorded as the measured width for the frame. The minimal width measured for an event was calculated as the minimal FWHM over the observation time times the pixel size of the iSIM setup.
Code availability
All code used in this project is available at https://github.com/LEB-EPFL/EDA
Author contributions
D.M., J.G. and S.M. conceived and designed the project. D.M., M.W. and S.M. supervised the project. D.M. built the iSIM microscope, collected the training data and performed the experiments on Cos-7 cells. D.M. and M.W. implemented the neural network for event detection. W.L.S. and D.M. implemented the EDA framework and performed data analysis. W.L.S. performed the experiments on C. crescentus. C.Z. prepared the C. crescentus strains and performed the sample preparation. D.M., W.L.S. and S.M. wrote the manuscript with contributions from all authors.
Competing interests
The authors declare that they have no conflict of interest.
Supplementary Information
Supplementary Note 1: Strains and plasmids
Caulobacter crescentus
The strains and plasmids used for this study are summarized in table 1 below. The wild-type strain (CB15N) was electroporated with the Pxyl::ftsZ-sfGFP plasmid and the PMT335-mScarletI plasmid sequentially to yield the dual-color strain. Notably, the DNA sequence of sfGFP and mScarletI used for this study was optimized for respective protein expression in C. crescentus. The two plasmids have been deposited at Addgene with the ID 174505 (Pxyl::ftsZ-sfGFP) and 174506 (PMT335-mScarletI).
Strains and plasmids used for this study
Supplementary Note 2: U-Net for the detection of divisions
The U-Net used for event detection in this project was trained with images of both Drp1 and the mitochondria outline. The Ground truth was calculated as discussed above. Frames are 10.4 µm × 10.4 µm.
The resulting network also gives reliable predictions of division precursor states upon inference on data from the instant SIM used for the implementation of EDA.
Scale bars: 1 μm
The different modes of imaging can clearly be seen in the bleaching curve represented by the signal-to-noise ratio calculated from the intensity inside the mitochondria compared to the signal outside of the mitochondria. For some parts with low frame rate, even a slight recovery of signal can be observed.
The strain used in this study can be synchronized to obtain a population of cells that are all at the beginning of their cell cycle. This leads to a time lag before the next divisions take place. As they are synchronized, many bacteria in the sample will then divide at the same time. We used EDA to sense the onset of divisions in the sample and adjust the imaging parameters in a way to optimally image during the divisions for high SNR and temporal resolution. a slow: 9 min, fast 3 min, thresholds 80, 100. b and c slow: 12min, fast 2 min, thresholds 90, 120. Scale bar: 1 µm
Top row: mitochondrial division as recorded with the slow fixed imaging rate. Thanks to EDA, additional frames are captured showing more detail of the dynamic of the event. Both the final constriction state and the fade of the Drp1 peak can be observed with higher temporal information. Scale bars: 1 µm
Mitochondria/C. crescentus label in white, neural network output in red. Events that triggered EDA where followed over time and the frame with the maximal neural network value was extracted as highlight. Some frames appear twice, if the neural network intensity was high enough to trigger EDA multiple times. Scale bar: 1 µm
Acknowledgements
We thank Hélène Perreten for technical support with cell culture and plasmid construction. We thank Laurent Casini and Justine Collier (University of Lausanne) for sharing plasmids and providing protocols for the bacterial experiments. Imaging data used for training the neural network in this publication were produced in collaboration with the Advanced Imaging Center, a facility jointly supported by the Gordon and Betty Moore Foundation and HHMI at HHMI’s Janelia Research Campus. We thank Lin Shao and Teng-Leong Chew at Janelia AIC for their help with SIM imaging.
