Abstract
Modern microscopy modalities create a data deluge with gigabytes of data generated each second, or terabytes per day. Storing and processing these data is a severe bottleneck, not fully alleviated by data compression. We argue that this is because images are processed as regular grids of pixels. To address the root of the problem, we here propose a content-adaptive representation of fluorescence microscopy images called the Adaptive Particle Representation (APR). The APR replaces the regular grid of pixels with particles positioned according to image content. This overcomes storage bottlenecks, as data compression does, but additionally overcomes memory and processing bottlenecks, since the APR can directly be used in processing without going back to pixels. We present the ideas, concepts, and algorithms of the APR and validate them using noisy 3D image data. We show that the APR represents the content of an image while maintaining image quality. We then show that the adaptivity of the APR provides orders of magnitude benefits across a range of image processing tasks. Therefore, the APR provides a simple, extendable, and efficient content-aware representation of images that relaxes current data and processing bottlenecks.
New developments in fluorescence microscopy (1–3), labeling chemistry (4), and genetics (5) provide the potential to capture and track biological structures at high resolution in both space and time. Such data is vital for understanding many spatiotemporal processes in biology (6). Unfortunately, fluorescence microscopes do not directly output the shapes and locations of objects through time. Instead, they produce raw data, potentially terabytes of 3D images (7), from which the desired spatiotemporal information must be extracted by image processing. Handling the large image data and extracting information from the raw microscopy images presents the main bottleneck (7–9). We propose that at the core of the problem is not the amount of information contained in the images, but how the data encodes this information – usually as a uniform grid of pixels. While data compression can alleviate storage issues, it does not reduce memory usage nor computational cost as all processing must still be done on the original, uncompressed data.
Processing bottlenecks are effectively avoided by the human visual system, which solves a similar problem of inferring object shapes and locations from photon accounts. In part, the human visual system achieves this by adaptively sampling the scene depending on its content (10), while adjusting to the dynamic range of intensity variations (11). This adaptive sampling works by selectively focusing the attention of the eyes on areas with potentially high information content (10). This selective focus then enables the efficient inference of information about the scene at a high effective resolution by directing the processing capacity of the retina and visual cortex. As in fluorescence microscopy, the information in different areas of a scene is not encoded in absolute intensity differences, but in relative differences compared to the local brightness. The human visual system maintains effective adaptive sampling across up to nine orders of magnitude of brightness levels (11) by using local gain control mechanisms that adjust to, and account for, changes in the dynamic range of intensity variations. Together, adaptation and local gain control enable the visual system to provide a high rate of information content using as little as 1 MB/s of data from the retina (12). In contrast, the rate of information in pixel representations of fluorescence microscopy images is much lower and is defined by the spatial and temporal resolution of the images rather than by their contents.
Inspired by the adaptive sampling and local gain control of the human visual system, we propose a novel representation of fluorescence microscopy images – the Adaptive Particle Representation (APR). The APR adaptively resamples an image, guided by local information content, while taking into account an effective local gain control. Figure 1A illustrates the basic idea of adaptive sampling. The top panel shows a pixel representation of a fluorescence image acquired from a specimen of Danio rerio, with labeled cell nuclei. The pixel representation places the same computational and storage costs in areas containing labeled cell nuclei and in areas with only background signal. This uniform sampling results in processing costs that are proportional to the spatial and temporal resolution of the acquisition, rather than the actual information content of the image. The main difficulty in adaptation, however, is to give equal importance to imaged structures across a wide range of intensity scales. This is achieved by local gain control as illustrated in Figure 1B. Without local gain control, adapting effectively to both bright and dim regions in the same image is not possible (centre left). The APR provides local gain control by guiding the adaptation by a Local Intensity Scale (center right). As seen in Figure 1B (right), this samples dim and bright objects at comparable resolution, giving them equal importance. Combining adaptive sampling and local gain control, the APR shares two key features of the human visual system to alleviate processing and storage bottlenecks in current fluorescence microscopy.
While the APR reduces storage costs, as data compression also does, it additionally overcomes memory and processing bottlenecks, since the APR can directly be used in processing without going back to pixels. Compression only alleviates storage costs, as the data need to be decompressed again for processing or visualization. The APR is therefore not a compression scheme, but an efficient image representation that can additionally then also be compressed. We posit that any image representation aiming to achieve this should fulfill the following representation criteria (RC):
RC1: It must guarantee a user-controllable representation accuracy for noise-free images and must not reduce the signal-to-noise ratio of noisy images.
RC2: Memory and computational cost of the representaion must be proportional to the information content of an image, and independent of the number of pixels.
RC3: It must be possible to rapidly convert a given pixel image into that representation with a computational cost at most proportional to the number of input pixels.
RC4: The representation must reduce the computational and memory cost of image-processing tasks without resorting to the original pixel representation.
None of the existing multi-resolution and adaptive sampling approaches meets all of these criteria, mainly because they were developed for different applications. However, most do share similar goals and use related concepts to achieve them. These approaches include super-pixels (13, 14), wavelet decompositions (15–17), error equidistribution methods (18–20), and dimensionality reduction (21, 22), as we discuss below. In contrast, we propose that the APR meets all of the above criteria. It provides a general framework combining concepts from wavelets, super-pixels, and equidistribution methods. Here, we present the APR for the first time and show that it fulfills all of the above representation criteria, making it an ideal candidate to replace pixel images in fluorescence microscopy.
The Adaptive Particle Representation
We illustrate the ideas and concepts behind the APR using a 1D Gaussian function as a didactic example. All of the concepts introduced extend to higher dimensions and to general continuous functions, as shown in the Supplement.
The APR takes an input pixel image and resamples it in a spatially adaptive way, representing it as a set of Particle Cells 𝒱 and intensity values stored at particle locations 𝒫. Particles, a generalization of pixels, are collocation points in space that carry properties, such as intensity, but are not restricted to sit on an evenly spaced grid and may have different sizes. The Particle Cells partition space and implicitly define the particle locations and a piecewise constant Implied Resolution Function R*(y) at all locations y in the image. Importantly, the Implied Resolution Function R*(y) also defines how the image is to be reconstructed at off-pixel and off-particle locations. It defines neighbor interactions between particles and specifies a local minimum resolution. From the APR, an image can be reconstructed at any location y by taking a non-negative weighted combination of particles that are within R*(y) distance of y. Formulated in this way, a pixel image is a set of particles placed on a regular grid with a constant Implied Resolution Function R*(y) = h, where h is the pixel size (as shown in Figure 2A, right). In contrast to uniform pixel representations, the APR adaptively represents an original image with particles whose locations, density, and sizes vary across the image (Figure 2A, left). As a result, computational and storage costs scale with the number of particles and no longer with the number of pixels.
Therefore, reducing the number of particles by adjusting the resolution to the image content, the APR can reduce storage and computational costs and increase the information-per-data ratio. This is optimally achieved by minimizing the number of particles required to represent a given image. However, the APR and any results computed from it must still reflect the content of the original pixel image.
Reconstruction Condition
For the APR to optimally represent a given image, the Implied Resolution Function should be set as large as possible at every location, while still guaranteeing that the image can be reconstructed within a user-specified relative error E scaled by the Local Intensity Scale σ(y). The Local Intensity Scale σ(y) is an estimate of the range of intensities present locally in the image. Considering an arbitrary Resolution Function R(y), we can formulate the problem as finding the largest R(y) everywhere that satisfies where Î(y) is the reconstructed intensity calculated by a non-negative weighted average over particles within R(y) distance of y. We call this the Reconstruction Condition and illustrate it in Figure 2B. For the 1D example shown in Figure 2, a constant local intensity scale σ(y) = 1 is used. We therefore focus solely on maximizing R(y) at each location. Maximizing R(y) minimizes , which is proportional to the locally required sampling density. Therefore, maximizing R(y) results in the minimum number of particles used. Unfortunately, finding the optimal R(y) that satisfies the Reconstruction Condition for arbitrary images requires a number of compute operations that scales with the square of the number of pixels N. This computational cost is prohibitive even for modestly sized images. We therefore propose two conservative restrictions on the problem and show that the optimal solution to the restricted problem can be computed with a total number of operations that is proportional to N.
APR Solution
Next, we outline the two problem restrictions, and how they are used to formulate an efficient linear-time algorithm for creating the APR.
Resolution Bound
The first restriction on the Resolution Function R(y) requires that for all original pixel locations y it satisfies the inequality where |y–y*| ≤ R(y), and . Here |∇I| is the magnitude of the image intensity gradient, which in 1D is and can be computed directly from the image. We call this inequality the Resolution Bound, and L(y) the Local Resolution Estimate. If we assume the continuous intensity distribution underlying the image to be differentiable everywhere and the Local Intensity Scale σ(y) to be sufficiently smooth (See SuppMat 2.1 for 1D and SuppMatEq 19 for nD), satisfying the Resolution Bound guarantees satisfying the Reconstruction Condition (See SuppMat 2 for 1D, and SuppMat 3 for nD). In Figure 2C, we illustrate that the Resolution Bound in 1D requires that a box centered at y of height R(y) and width 2R(y) does not intersect anywhere with the graph of L(y). Since the Resolution Bound represents a tighter bound than the Reconstruction Condition, the optimal solution to the Resolution Bound Rb(y) is always less than or equal to the optimal solution to the Reconstruction Condition Rc(y), therefore providing the same or a higher image representation accuracy. The dashed lines in Figure 2D illustrates this for the 1D example. As mentioned above, solving for the optimal Resolution Function has a worst-case complexity in O(N 2). However, we show next that the Resolution Bound can be found optimally with a linear complexity in N if we re-strict the Resolution Function to be composed of square blocks.
Finding the Resolution Function with Particle cells
The second restriction is that the blocks constituting the Resolution Function must have edge lengths that are powers of 1/2 of the image edge length. The piecewise constant Resolution Function defined by the uppermost edges of these blocks is called the Implied Resolution Function R*(y) and is shown in black in Figure 2D. The blocks we call Particle Cells. They have sides of length , where |Ω| is the edge length of the image, measured in pixels. The number l is a positive integer we call the Particle Cell Level. Each Particle Cell ci, l, is therefore uniquely determined by its level l and location i. Figure 2E illustrates these definitions for a single Particle Cell (See SuppMat 4 for the nD formal definition). The size of the blocks on the lowest level is half the size of the image (lmin = 1), and the highest level of resolution lmax contains boxes the size of the original pixels. For image edge lengths that are not powers of 2, |Ω| is rounded upwards to the nearest power of two.
Using these two restrictions, the problem of finding the optimal Resolution Function can be reduced to finding the smallest set V of blocks that defines an Implied Resolution Function R*(y) that satisfies the Resolution Bound (SuppMat 4.1). We call this set V of Particle Cells the Optimal Valid Particle Cell (OVPC) set.
In order to construct an algorithm that efficiently finds the OVPC set for a given Local Resolution Estimate L(y), we first formulate the Resolution Bound in terms of Particle Cells. This formulation requires arranging the set C of all possible Particle Cells ci, l by level l and location i in a tree structure, as shown in Figure 2F. In 1D this is a binary tree, in 2D a quad-tree, and in 3D an oct-tree. When arranged as a tree structure, we can naturally define children and neighbor relationships between Particle Cells, as respectively shown in green and blue in the example. Also, we define the descendants of a Particle Cell as the set of all children and children’s children up to the maximum resolution level lmax. Given these definitions, the Local Resolution Estimate L(y) can be represented as a set of Particle Cells ℒ by iterating over each pixel y*, and adding the Particle Cell with level and location to ℒ if it is not already in ℒ (assuming the lower-left boundary of the image is at zero). Figure 2G illustrates how ℒ relates to L(y), with ℒ also represented in Figure 2F in the tree structure. We call this set of Particle Cells the Local Particle Cell (LPC) set ℒ (See SuppMat 4.2).
We can then represent the Resolution Bound in terms of ℒ. A set of Particle Cells 𝒱will define an Implied Resolution Function that satisfies the Resolution Bound for L(y), if and only if the following statement is true: for every Particle Cell in 𝒱, none of its descendants, or neighbors’ descendants, are in the LPC set ℒ (SuppTheorem 1). We call any set of Particle Cells satisfying this statement valid. The OVPC set 𝒱is then defined as the valid set for which replacing any combination of Particle Cells with larger Particle Cells would result in 𝒱 no longer being valid (SuppTheorem 2).
Pulling Scheme
We present an efficient algorithm for finding the OVPC set 𝒱 called the Pulling Scheme. The name is motivated by the way a single Particle Cell in ℒ pulls the resolution function down to enforce smaller Particle Cells across the image. The Pulling Scheme finds the OVPC set 𝒱 directly, without explicitly checking for validity or optimality. The result is by construction guaranteed to be valid and optimal. In order to derive the algorithm, we leverage three properties of OVPC sets:
Predictable and self-similar structure: Neighboring Particle Cells never differ by more than by one level and are arranged in a fixed pattern around the smallest Particle Cells in the set. This local structure is independent of absolute level l and endows the set with a self-similar structure. Using this structural feature, the OVPC set 𝒱 for a LPC set ℒ with only one Particle Cell ci, l can be generated directly for any i and l.
Separability: We can find the OVPC set given a LPC set ℒ by considering each cell in ℒ separately and then combining the smallest Particle Cells from all sets that cover the image (see SuppLemma 1). Figure 3A illustrates this separability property.
Redundancy: The redundancy property tells us that when constructing 𝒱 we can ignore all Particle Cells in ℒ that have descendants in ℒ. This is because descendants provide equal or tighter constraints on the resolution function than their parent Particle Cells (see SuppLemma. 2 for the proof).
These properties enable us to efficiently construct 𝒱 by propagating solutions from individual Particle Cells in ℒ, one level at a time, starting from the highest level lmax) of the smallest Particle Cells in ℒ. Here we use a simple implementation that explicitly represents all possible Particle Cells in an image pyramid structure1. The Pulling Scheme is summarized in Supplementary Algorithm 1, and Figure 3B illustrates the steps for each level. SuppMat 5.5 and SuppMat 12.5 provide additional details. The computational complexity of the neighbor operations in the algorithm scales with the number of Particle Cells in 𝒱 and guarantees validity and optimality by construction. Computing the OVPC set 𝒱 using the Pulling Scheme incurs a computational cost that is at most proportional to the number of pixels N.
The computational and memory performance of the Pulling Scheme is reduced by a factor of 2d, where d is the image dimensionality, while obtaining the same solution and algorithm as above, using the Equivalence Optimization (See SuppMat 5.4 and SuppMat 5.7). This restricts calculations on the full image to filtering operations for the gradient magnitude and greatly improves memory and computational efficiency of the method. A second optimization restricts the neighborhood of particle cells to further reduce computational cost, as described in SuppMat 5.6. We use both optimizations for results presented in this paper.
Placing the Particles 𝒫
Given the Implied Resolution Function computed by the Pulling Scheme, the last step of forming the APR is to determine the locations of the particles 𝒫. This must be done such that around each pixel location y there is at least one particle within a distance of R*(y). This is most easily achieved by placing one particle at the center of each Particle Cell in 𝒱. Specifically, for each Particle Cell ci, l in 𝒱, we add a particle p to 𝒫 with location . For each particle p we store the image intensity at that location Ip = I(yp) interpolated from the original pixels as described in SuppMat 6. This way of arranging the particles has the advantage that the particle positions do not need to be stored, as they can be directly computed from 𝒱.
Forming the APR= {𝒱, 𝒫}
In Figure 3B we outline the steps required to form the APR from an input image. The APR can be stored as the combination of {𝒱, 𝒫}. We represent the OVPC set 𝒱 by storing the integer level l and the integer location i for each Particle Cell. 𝒫 then defines the Implied Resolution Function R*(y) for all y in the image. The second component, the particle set 𝒫, stores the properties of each particle p, i.e., its intensity and type. Since the particle positions do not need to be stored, the APR is efficiently represented in memory.
Practical Considerations
Determining L(y) requires computing the intensity gradient ∇I over the input image. In practice, the pixel intensities are noisy, which leads to uncertainty in the computed L(y). In SuppMat 7, we provide theoretical results how this uncertainty imposes a lower bound on the achievable E.
Data structures
Appropriate data structures must be used to efficiently store and process on the APR. Ideally, these structures allow direct memory access at low overhead. Here, we propose a multi-level data structure for the APR, as described in SuppMat 17. Each APR level is encoded similar to sparse matrix schemes. These data structures efficiently encode 𝒱 and 𝒫 by explicitly encoding only one spatial coordinate per Particle Cell, while allowing random access. We call this data structure the Sparse APR (SA). It relies on storing one red-black tree per x, z and level, caching access information for contiguous blocks of Particle Cells. When storing image intensity using 16 bits, the SA data structure requires approximately 50% more memory than the intensities alone.
3D Fluorescence APR Implementation
We assess the properties of the APR for noisy 3D images. Figure 3B illustrates the main steps of the implementation using a 2D example. Additionally, results without noise in 1D are given in SuppMat 11 (code available from github.com/cheesema/APR 1D demo).
When implementing the APR, three design choices have to be made. First, one has to decide how to calculate the gradient magnitude |∇I(y)|. Second, one has to decide how to compute the Local Intensity Scale σ(y). Third, one has to decide how to interpolate the image intensity at particle locations Ip = I(yp). Full details and descriptions of parameters given in SuppMat 12.
To calculate the gradient magnitude over the input image we use smoothing cubic B-Splines (25), which provide robust gradient estimation in the presence of noise. They require the setting a smoothing parameter ? depending on the noise level.
For the Local Intensity Scale σ(y), we use a smooth estimate of the local dynamic range of the image, as described in SuppMat 12.3. This form of the local intensity scale accounts for variations in the intensities of labeled objects, similar to gain control in the human visual systems. We ensure that σ is sufficiently smooth (See SuppMat 4.4) by computing it over the image downsampled by a factor of two. Examples are shown in Figures 1B and 3B. The size of the smoothing window is set by a coarse estimate of the standard deviation of the point-spread function (PSF) of the microscope. Further, a minimum threshold is introduced to prevent resolving background noise.
We find that the method is stable with respect to the choice of these parameters and a discussion on parameter selection for real datasets and their impact on the APR is given in Suppmat 13 and parameters used for real datasets are given in STable 3.
Two methods are combined to interpolate pixel intensities to particle locations: for particles in Particle Cells at pixel resolution, the intensities are directly copied from the respective pixels, while for particles in larger particle cells, we assign the average intensity of all pixels in that Particle Cell (26).
For benchmarking we need a method for reconstructing a pixel image from the APR. A pixel image satisfying the Reconstruction Bound can be reconstructed from the APR using any non-negative weighted average of particles within R*(y) of pixel y. In SuppMat 10 we discuss possible weight choices, providing examples of smooth, piecewise constant, and worst-case reconstructions. For displaying figures and benchmarking, unless otherwise stated, we use the piecewise constant reconstruction. This reconstruction sets the pixels inside every Particle Cell equal to the intensity of the particle in that cell and thus has the best computational efficiency.
All design decisions have been made to optimize robustness against imaging noise and computational efficiency.
Validation
All benchmarks use the open-source C++ APR software library libAPR (github.com/cheesema/LibAPR) compiled with with gcc 5.4.0 and OpenMP shared-memory parallelism on a 10-core Intel Xeon E5-2660 v3 (25 MB cache, 2.60 GHz, 64 GB RAM) running Ubuntu Linux 16.04. SuppMat 15 provides a detailed description of each benchmark and the parameters used.
Benchmarks on synthetic data
We first assess the performance of the APR using synthetic benchmark data. SuppMat 14 and SFigure 22 outline the synthetic data generation pipeline. The key advantage of synthetic data is that all relevant image parameters can be varied and the ground-truth image is known. Synthetic images are generated by placing a number of blurred objects into the image domain and corrupting with modulatory Poisson noise. We study the influence of image size, content, and noise level on the performance of the APR. Spherical objects are used for simplicity unless otherwise indicated.
Reconstruction Bound
We experimentally confirm that the APR satisfies the Reconstruction Condition in the absence of noise. Figure 4A shows the empirical relative error for increasing imposed error bounds E. In all cases, E* < E (dashed line), as required by the Reconstruction Condition. We find that the number of particles used by the APR to represent the image decreases with increasing E (right axis). The results are unchanged when using more complex objects than spheres (SFigure 25) and using different reconstruction methods (SFigure 25). Figure 4C provides examples of the quality of APR reconstruction at different levels of E compared to ground truth. In the absence of noise, the APR satisfies the Reconstruction Condition everywhere, guaranteeing a reconstruction error below the user-specified threshold, fulfilling the first part of RC1.
Robustness against noise
In real applications, images are corrupted by noise. We find that the introduction of noise introduces a lower limit on the relative error E* that can be achieved (see first plot in SFigure 26). This observation agrees with theoretical analysis (SuppMat 7.3). This lower bound is entirely due to the noise in the pixel intensity values, while the adaptation of the Implied Resolution Function R*(y) is robust to noise. This is demonstrated in the second plot in SFigure 26, where noisy particle intensities are replaced with ground-truth values for the reconstruction step. Adaptation is still done on the noisy pixel data. Again, E* can be made arbitrarily small, indicating that the construction of the APR is robust against imaging noise. This result also agrees with the theoretical analysis of the impact of errors in L(y) on the Implied Resolution Function (SuppMat 7.2).
To understand how to best set E in the presence of noise, we compute the observed peak signal-to-noise ratio (PSNR) of the reconstructed image and compare with the PSNR of the original image. Figure 4B shows that decreasing E to zero does not maximize the PSNR. Instead, for medium to high quality input images, the PSNR is highest between an E of 0.08 and 0.15. For low-quality input images, we find a monotonic relationship between the PSNR and E, as de-noising from downsampling dominates. Also, for E < 0.2 the reconstruction error is always less than the noise in the input image, reflected in a PSNR ratio greater than one. Therefore, for noisy images with medium to high quality, there is an optimal range for E between 0.08 and 0.15. In this range, the reconstruction errors are less than the imaging noise, and the signal-to-noise ratio of the APR is better than that of the input pixel image, fulfilling also the second part of RC1.
Response to image content
In Figure 4D we show how the APR adapts to image content. This adaptation is manifest in the linear relationship between the number of objects (spheres) randomly placed in the image and the number of particles used by the APR (right axis). Adaption is linear despite the brightness of the objects randomly varying over an order of magnitude (see SuppMat 15.4). Image quality is maintained throughout (left axis). Figure 4C shows an example of a mediumquality input image and its APR reconstruction. Figure 4E shows that the number of particles used by the APR to represent a fixed number of objects becomes independent of image size. Also, if pixel resolution and image size are increased proportionally, the APR approaches a constant number of particles (SFigure 27). These results show that the APR adapts proportional to image content, independent of the number of pixels, fulfilling RC2.
Evaluation of the Local Intensity Scale
So far, we have not directly assessed the validity of the Local Intensity Scale σ. In order to do this, we need a groundtruth reference. In SuppMat 14.5 we introduce the perfect APR, and the Ideal Local Intensity Scale σideal that can be calculated for synthetic data. This ground-truth representation is then used to benchmark the APR.
The results in STables 1 and 2 show that the local intensity scale we use is effective over wide range of scenarios. However, for crowded images with large contrast variations (two orders of magnitude or more), we find that the Local Intensity Scale over-estimates the dynamic range of dim regions that are close to bright regions. This effect is most pronounced in high-quality images, where alternative formulations of the Local Intensity Scale could provide better results.
Computational cost
Due to the adaptivity of the APR, its computational cost depends on image content through the number of particles, and not on the input image size N. For a given input image, we define the Computational Ratio (CR) as:
We assess the performance of the APR for synthetic images with numbers of objects roughly corresponding to CR = 5, 20, 100, representing high, medium, and low complexity images (SFigure 28, SuppMat 16.1). The results are given in Table 1. The APR achieved effective CR values of 5.63, 19.7, and 93.9, respectively.
Benchmarks on real data
We present results for a corpus of 19 exemplar volumetric fluorescence microscopy datasets of different content and imaging modalities, ranging in size from 160 MB to 4 GB. The datasets and parameters used are described in STable 3&4 and summary statistics in Table 1. SFigure 29 shows a cross-section of the APR for exemplar dataset 7 and SVideo 1 illustrates the Resolution Function and reconstruction for exemplar dataset 1.
Memory requirements
Calculation of the APR from an image requires approximately 2.7 times (for 16-bit images) the size of the original image in memory. Further, the maximum size is only limited by available main memory (RAM) of the machine and by the ability to globally index the particles using an unsigned 64-bit integer. Our pipeline has been successfully tested on datasets exceeding 100 GB.
Execution time
On our benchmark system, we find linear scaling in N and an average data rate of 507 MB/s for transforming images to their APR. This rate corresponds to 3.9 seconds to form the APR from an input image of size N = 10003. On the exemplars, execution times range from 0.37 seconds to 8.14 seconds, with an average of 3.65 seconds. Table 1 summarizes the results. We find the following distribution of computation time: the Pulling Scheme on average takes less than 3.5% of the total time while the computation of the gradient magnitude using smoothing B-splines dominates the execution time, taking up to 59% of the total time. For details see SuppMat. 18.
The pipeline shows efficient parallel scaling (Amdahl’s Law, parallel fraction = 0.95) on up to 47 cores, achieving data rates of up to 1400 MB/second. This enables real-time conversion of images to the APR, as it is faster than the acquisition rate of microscopes (28, 29).
We conclude that images can be rapidly converted into the APR with a cost that scales at most linearly with image size N, fulfilling RC3.
Storage requirements
For the fixed-CR datasets, we observe an average Memory Compression Ratio 2 of 1.4 times the CR. The median MCR of the exemplars is 36.8, and the mean is 129.5. This corresponds to an average size of the input images of 1.87 GB and the compressed APR of 51 MB. Table 1 summarizes the results, additionally showing MCR for pixel images stored using lossless pbzip2 compression.
In the APR files, on average 89% of the bytes are used to store the particle intensities, implying that the overhead introduced by the APR data structures is 11% on average. These compression ratios are comparable to custom lossy compression methods designed specifically for storing of fluorescence microscopy images (27, 30). In addition, the APR particle intensities can be compressed in a lossy manner. As an example, in Table 1 we also give the MCR using the within-noise-level (WNL) compression algorithm (27) applied to particles at pixel resolution, achieving an additional 1.4…4 compression factor.
The APR can be efficiently compressed with a file size proportional to the image content, fulfilling RC2. Unlike compression techniques, the APR is an image representation that can be leveraged in downstream processing tasks without going back to the original pixel image.
Image Processing on the APR
We show how the APR reduces the memory and computational cost of downstream image-processing tasks (RC4). Once we have transformed the input image into an APR, the input image is no longer needed. All processing, storage, and visualization can be done directly on the APR.
Image-processing methods are always developed using a certain interpretation of images. Just like pixels, one can also interpret and use the APR in different ways depending on the processing task. These interpretations align with those commonly used in pixel-based processing. Figure 5A-D outlines the four main interpretations of the APR.
Performance metrics
The APR can accelerate existing algorithms in two ways: First, by decreasing the total processing time through reducing the number of operations that have to be executed. Second, by reducing the amount of memory required to run the algorithm. The relative importance of the two, and the degree of reduction, depends on the specific algorithm and its implementation. We use quantitative metrics to evaluate the improvements for different algorithms and input images.
The first evaluation metric relates to the computational performance of the algorithm. For a given algorithm and implementation, we define the speed-up (SU) as:
It is insightful to relate the SU to the CR by SU = CR * (Pixel-Particle Speed Ratio) (PP), where PP = (Time to compute the operation on one pixel)/(Time to compute the operation on one particle). The value of PP depends on many factors, including memory access patterns, data structures, hardware, and the absolute size of the data in memory. Consequently, even for a given algorithm running on defined hardware, the PP is a function of the input image size N. Therefore, for tasks where PP<1, as in most low-level vision tasks, there is a minimum value of CR for which the algorithm is faster on the APR than on pixels.
For an algorithm run on a pixel image, in most cases the Memory Cost (MC) in bytes scales linearly with the number of pixels N and the number of algorithm variables, as MC=(Number of variables)*(Data type in Bytes)*N. The memory cost of the APR where Np is the number of particles, and the cost of the data structure per particle depends on N. We find an estimated average of 8 bits per particles overhead for the Sparse APR data structure. As the number of algorithm variables increases, the overhead of the APR is amortized so that the reduction in memory cost approaches the CR.
Image Processing Performance Benchmarks
We analyze two low-level and one high-level imageprocessing task. These are neighbor access and filtering as low-level tasks, and image segmentation as highlevel task. The low-level tasks represent a lower bound on the benefits of the APR due to their simple operations and access patterns, which are best suited for processing on pixels. The segmentation task in contrast provides a representative practical example of microscopy image analysis.
For these three benchmarks, we provide results for the computational and memory metrics for three fixed-CR datasets with input images from N = 2003 up to N = 10003, and for all real-world exemplar datasets. The results of all benchmarks are summarized in Table 2. SuppMat 21 describes the benchmark protocols.
Neighbor Access
For each pixel or particle, the task involves averaging the intensities of all face-connected neighbors (see SuppMat 21.1 for details). In the APR, neighbors are defined by the particle graph, as shown in Figure 5C and described in SuppMat 20. We benchmark two forms of neighbor access: Linear iteration loops over all neighbors in sequential order. Random access visits neighbors in random order, irrespective of how they are stored in memory.
For linear iteration, the APR shows low speed-ups. It is even slower than pixel operations for images with CR=5 and for four of the exemplar datasets (Table 2, group 1). This is because linear iteration is optimally suited to pixel images. However, the APR provides consistently higher speed-ups for random neighbor access, especially for high CRs. This is likely due to the smaller overall size of the APR improving cache efficiency.
The total memory cost of the APR reflects the CR of the dataset. This provides significant memory cost reductions across all benchmark datasets for both the linear and random neighbor access patterns.
Image filtering
We consider the ftask of filtering the image with a Gaussian blur kernel (see SuppMat 21.2). We exploit the separability of the kernel and perform three consecutive filtering steps using 1D filters in each direction. On the APR, this requires locally evaluating the function reconstruction. For simplicity, we use the piecewise constant reconstruction method (see SuppMat 10). The benchmark results are shown in Table 2, group 3. Directly filtering the APR consistently outperforms the pixel-based pipeline, both in terms of memory cost and execution time.
In SuppMat 21.2.2 we analyze the results in detail and find that the APR is most appropriate if the filtering result looks similar to the original image, such that the same set of content-adapted particles is also suitable to represent the filtered image. SFigure 35 illustrates this, showing how for a small blur the APR filter has higher PSNR than the pixel filter. For larger blurs this is reversed, because the specific APR adapted to the input image is no longer suitable to represent the filtered image. Care must be taken when designing algorithms, as not all approaches are equally suited to the APR.
Image segmentation
We perform binary image segmentation using graph cuts, using the method and implementation of Ref. (32) to compute the optimal foreground/background segmentation for both APR and pixel images. When computing the cut energies, we exploit the additional information provided by the particle cell level, type, and local min-max range. To allow direct comparison with the pixel-based segmentation, we interpolate all energies calculated on the APR to pixels and determine the cuts over the pixel image using the same energies. For both APR and pixel images, a face-connected neighbourhood graph is used. Given the energy calculations are identical, we benchmark the execution time and memory cost of the graph-cut solver. The results are shown in Table 2, group 4. For the APR we find speed-ups directly reflecting the CR.
Using the APR, all examplar images can be segmented without problems, illustrating the benefits of the reduced memory cost of the APR, while pixel images can only be segmented for sizes N ≤ 5503 on our benchmark machine with 64 GB RAM.
We validate the APR segmentations by comparing both the APR and pixel-based segmentations to ground truth using the Dice coefficient (36). Across datasets, we find that the Dice coefficients are not statistically significantly different (p-value: 0.92, Welch’s t-test). We provide a representative example in SVideo 2 and show a 3D rendering of a segmentation in Figure 6E. For more details, see SuppMat 21.3.
Novel Algorithms
The APR provides additional information about the image that is not contained in pixel representations. This information can be exploited in image-processing algorithms, as illustrated in the segmentation example above. In addition, it can also be used to design entirely novel, APR-specific algorithms, as demonstrated in the following example.
Adaptive APR filter
We define a discrete filter over neighboring particles in the APR particle graph. Since the distance between neighboring particles varies across the image depending on image content, this amounts to spatially adaptive filtering with the filter size automatically adjusting to the content of the image. On the APR, this only requires linear neighbor iteration, while an adaptive pixel implementation would be significantly more complex.
SuppMat 21.4 describes the adaptive APR filter in detail. SFigure 37 shows synthetic results for an adaptive blurring filter, and SFigure 38 for a filter that adaptively estimates the intensity gradient magnitude. In both examples, the adaptive APR-filtered results have higher PSNR than results from corresponding non-adaptive pixel filters, also demonstrated in Figure 6C.
Visualization
Images represented using the APR can directly be visualized without going back to pixels. The APR image can be visualized using both traditional and novel visualization methods. We provide examples of the following visualization methods:
Visualization by slice, Figure 6A and SVideo 1 show examples of a slice-wise APR reconstruction in comparison with the pixel image.
Raycasting for direct 3D visualization of an APR. Figure 6B and SVideo 3 show a perspective maximumintensity projection in comparison with the same raycast of the original pixel image. The resulting visualizations are visually indistinguishable. APR raycasting only requires storing and computing on the APR, reducing memory and computational costs proportionally to the CR of the image, making direct visualisation of very large datasets possible.
Particle rendering, where we directly visualize the particles of the APR as glyphs (see Figure 3 and Figure 5; Figure 6D and SVideo 4&5 additionally show examples of particle renderings in 3D using open-source rendering toolkit scenery (35)).
For more details on visualisation, see SuppMat 21.5.
Image Processing Summary
Across all benchmarks and exemplar datasets other than the worst-case example of linear neighbor access, processing directly on the APR resulted in smaller execution times and memory costs. In most cases, the reductions are directly proportional to the computational ratio (CR), hence fulfilling RC4. Moreover, in the examples of visualization and segmentation, the memory cost reduction of the APR enabled processing of data sets that would not otherwise have been possible on our benchmark machine. The APR has a range of interpretations that align with those of pixel images, allowing direct application of established image-analysis frameworks to the APR.
In addition, we highlight that the APR may simplify processing tasks by providing additional information about the structure of the image through the Particle Cell level and type. This structural information can be leveraged in existing algorithms, as shown for segmentation, or it can be used to design novel algorithms, such as the adaptive APR filter and APR ray-casting visualization.
Discussion and Conclusion
We have introduced a novel content-adaptive image representation for fluorescence microscopy, the Adaptive Particle Representation (APR). The APR is inspired by how the human visual system effectively avoids the data and processing bottlenecks that plague modern fluorescence microscopy, particularly for 3D imaging. The APR combines aspects of previous adaptive-resolution methods, including wavelets, super-pixels, and equidistribution principles in a way that fulfills all representation criteria set out in the introduction. The APR is computationally efficient, suited for real-time applications at acquisition speed, and easy to implement.
We presented the ideas and concepts of the APR in 1D for ease of illustration, with all naturally extending to higher dimensions. The APR resamples an image by adapting a set of Particle Cells 𝒱 and a set of particles 𝒫 to the content of an image, taking into account the Local Intensity Scale σ similar to gain control in the human visual system. The main theoretical and algorithmic contribution that made this possible with a computational cost that scales linearly with image contents is the Pulling Scheme. The Pulling Scheme guarantees sub-optimal image representations within user-specified relative intensity deviations.
We verified accuracy and performance of the APR using synthetic benchmark images. The analysis showed that all theoretical results hold in practice, and that the number of particles used by the APR scales with image content while maintaining image quality (RC1). Further, we showed that although image noise places a limit on representation accuracy, there exists an optimal range for the relative error threshold E. In this range, the reconstruction error for noisy images is always well within the imaging noise level (RC1). Moreover, we found that the number of particles is independent of the original image size, with computational and memory costs of the APR proportional to the information content of the image (RC2). We showed how pixel images can rapidly be transformed to the APR, and efficiently stored both in memory and in files (RC3). We have demonstrated that the APR benefits both in terms of execution time and memory requirements can be leveraged for a range of image-processing tasks without ever returning to a pixel image (RC4). Finally, we showed how the adaptive sampling and structure of the APR inspires the development of novel, contentadaptive image-processing algorithms.
Taken together, the APR meets all four Representation Criteria (RC) set out in the introduction. We believe that the gains of the APR will in many cases be sufficient to alleviate the current processing bottlenecks. In particular, image-processing pipelines using the APR would be well suited for high-throughput experiments and real-time processing, e.g., in smart microscopes (9, 37). However, the APR is sub-optimal with respect to the number of particles used. This sub-optimality results from the conservative limiting assumptions required to derive the efficient Pulling Scheme. It is easily seen by the fact that the APR particle properties could be represented by a Haar wavelet transform (33) with nonzero coefficients whose number is either equal to, or less than, the number of particles in the APR while allowing exact reconstruction of the APR particle properties.
The use of adaptive representations of images (38–40) and its motivation by the human visual system (13, 41) are not new. The APR shares several principles and ideas with established adaptive representations. The Resolution Function R(y) of the APR, e.g., is related to the oracle adaptive regression method (42) and the derivation and form of the Resolution Bound are related to ideas originally introduced in equidistribution methods for splines (18, 43, 44), which also inspired the work here (20). The Reconstruction Condition for a constant Local Intensity Scale relates to infinity norm adaptation (45) for wavelet thresholding in adaptive surface representations. Further, the use of a powers-of-two decomposition of the domain is central to many adaptiveresolution methods (26, 33, 34, 46) and its use here was particularly inspired by Ref. (47). Despite its similarity to existing methods, the APR uniquely fulfills all representation criteria and extends many of the previous concepts. Core novelties include the spatially varying Local Intensity Scale, the fact that the APR works with a wider class of reconstruction methods, provides theoretical bounds on arbitrary derivatives of the represented function, and enables integration of additional spatial constraints with no changes to the Particle Cell formulation or the Pulling Scheme algorithm.
Outlook
The APR has the potential to completely replace pixelbased image-processing pipelines for the next generation of fluorescence microscopes. We envision that the APR is immediately formed, possibly after image enhancement (48), on the acquisition computer or even on the camera itself. Following this, all data transfer, storage, visualization, and processing can be done using the APR, providing memory and computational gains across all tasks. Although, there are also contexts when the exact pixel noise distribution of the original image conveys information, and therefore the APR is not appropriate. In addition, the realization of such pipelines requires further algorithm and software development including integration with current microscope systems, image databases (49), and image-processing tools (50).
Here, we presented a particular realization of an APR pipeline. We foresee alternative pipelines, e.g., using deep learning approaches (51) to provide improved estimation of the Local Intensity Scale, the image intensity gradient, and the smooth image reconstruction. Just as in space, the APR can also be used to adaptively sample time. Such temporal adaptation can lead to a multiplicative reduction in memory and computational costs compared to those presented here, allowing even faster APR computations. Further, the APR can be extended to allow for anisotropic adaptation using rectangular particle cells and anisotropic particle distributions within each cell.
Given the wide success of adaptive representations in scientific computing, the unique features of the APR could be useful also in non-imaging applications. This includes applications to time-series data, where the APR could provide an adaptive regression method (42), and to surface representation in computer graphics (45). Further, the APR could be used in numerical simulations for efficient mesh generation or as an adaptive mesh-free collocation method for numerically solving partial differential equations (20, 52–54).