ABSTRACT
Organoids are three-dimensional in vitro tissue models that closely represent the native heterogeneity, microanatomy, and functionality of an organ or diseased tissue. Analysis of organoid morphology, growth, and drug response is challenging due to the diversity in shape and size of organoids, movement through focal planes, and limited options for live-cell staining. Here, we present OrganoID, an open-source image analysis platform that automatically recognizes, labels, and tracks single organoids in brightfield and phase-contrast microscopy. The platform identifies organoid morphology pixel by pixel without the need for fluorescence or transgenic labeling and accurately analyzes a wide range of organoid types in time-lapse microscopy experiments. OrganoID uses a modified u-net neural network with minimal feature depth to encourage model generalization and allow fast execution. The network was trained on images of human pancreatic cancer organoids and was validated on images from pancreatic, lung, colon, and adenoid cystic carcinoma organoids with a mean intersection-over-union of 0.76. OrganoID measurements of organoid count and individual area concurred with manual measurements at 96% and 95% agreement respectively. Tracking accuracy remained above 89% over the duration of a four-day validation experiment. Automated single-organoid morphology analysis of a dose-response experiment identified significantly different organoid circularity after exposure to different concentrations of gemcitabine. The OrganoID platform enables straightforward, detailed, and accurate analysis of organoid images to accelerate the use of organoids as physiologically relevant models in high-throughput research.
INTRODUCTION
Three-dimensional (3D) cell culture systems serve as more physiologically relevant in vitro models than traditional human cell monolayers for both basic and clinical applications1. Organoids are multicellular 3D structures derived from primary or stem cells that are embedded into a biological matrix to create an extracellular environment that provides structural support and key growth factors. Organoids are a particularly useful culture system, as they closely recapitulate cellular heterogeneity, structural morphology, and organ-specific functions of a variety of tissues2–5. Live-cell imaging can reveal important organoid dynamics, such as growth, apoptosis/necrosis, and movement, which can reflect physiological and pathological behaviors and drug responses. While organoids have been successfully used to investigate important phenomena that might be obscured in simpler models, their use in data-intensive applications, such as high-throughput screening, has been limited.
A major challenge for high-throughput organoid experiments is response measurement and data analysis, which must be performed for a large number of microscopy images. Image analysis is particularly difficult for organoid experiments due to their movement across focal planes and variability in organoid size and shape between different tissue types, within the same tissue type, and within the same single culture sample6,7. Cells can be genetically modified to express fluorescent proteins8–10 that aid in image segmentation and tracking, yet the transformation process increases experimental time and complexity and may alter cellular dynamics from original samples11,12. Small fluorescent dyes can penetrate live cell membranes to stain intracellular structures, however this approach is inadequate for organoid culture due to cumulative toxicity over longer growth times and limited diffusion through the hydrogel matrix13. There is a critical need for an automated image analysis tool that can measure detailed organoid responses without the use of potentially toxic or confounding live-cell fluorescence techniques.
A number of software tools have been developed to automate the process of organoid image analysis. These platforms use conventional image processing methods, such as adaptive thresholding and mathematical morphology14, or convolutional neural networks15,16 to identify organoids in sequences of microscopy images. Despite their advantages, most existing platforms require cellular nuclei to be transgenically labeled16, which increases experiment time and complexity and may modify cellular dynamics, require manual parameter tuning for each image14, or are limited to bounding-box detection15, which fails to capture potentially useful morphological information. Changes in organoid shape, such as spiking or blebbing, can reveal important responses to external stimuli organoids and might be missed with bounding-box measurements30. Importantly, existing platforms were developed for analysis of organoids derived from a single type of tissue and for images obtained with one specific geometric configuration.
To address these challenges, we have developed a software platform, OrganoID, that can identify and track organoids derived from a wide range of tissue types, pixel by pixel, in both brightfield and phase-contrast microscopy images. OrganoID consists of (i) a convolutional neural network, which detects organoids in microscopy images, (ii) an identification module, which resolves contours to label individual organoids, and (iii) a tracking module, which follows identified organoids in time-lapse imaging experiments. Most importantly, OrganoID accurately segments and tracks a wide range of organoid types, including those derived from pancreatic ductal adenocarcinoma, adenoid cystic carcinoma, and lung and colon epithelia. OrganoID is publicly available both as source code and a standalone executable. The distribution package also includes sample datasets and detailed usage instructions and examples. The OrganoID software overcomes a major hurdle to organoid image analysis and supports wider integration of the organoid model into high-throughput applications.
RESULTS
A robust convolutional neural network for pixel-by-pixel organoid detection
We developed a deep learning-based image analysis pipeline, OrganoID, that recognizes and tracks individual organoids, pixel by pixel, for bulk and single-organoid analysis in brightfield and phase-contrast microscopy images (Figure 1a). The platform employs a convolutional neural network to transform microscopy images into organoid probability images, where brightness values represent the network belief that an organoid is present at a given pixel. The network structure was derived from the widely successful u-net approach to image segmentation9 (Figure 1b). The u-net approach first passes each image through a contracting series of multidimensional convolutions and maximum filters to extract a set of deep feature maps that describe the image at various levels of detail and contexts, such as edges, shapes, and textures. The feature maps are then passed through an expanding series of transposed convolutions, which learn to localize the features and assemble a final output. The OrganoID neural network follows a u-net structure and was optimized to require far fewer feature channels than the original implementation. Model simplification can limit overfitting and minimizes the amount of memory and computational power required to use the network in the final distribution. Additionally, all hidden convolutional layers were set to compute outputs with the exponential linear unit (ELU) activation function18, which has demonstrated higher accuracy than the rectified linear unit (ReLU) and avoids vanishing gradient problems during network training. Neurons in the final convolutional layer were set with a sigmoid activation function to produce a normalized output that corresponds to the probability that an organoid is present at each pixel in the original image.
An original dataset of 66 brightfield and phase-contrast microscopy images of organoids were manually segmented to produce black-and-white ground truth images for network training and validation (Figure 1c). Each image featured 5 to 50 organoids derived from human pancreatic ductal adenocarcinoma (PDAC) samples from two different patients. Organoids were either grown on a standard tissue culture plate or on our previously published microfluidic organoid platform19. To teach the network that segmentations should be independent of imaging orientation, field-of-view, lens distortion, and other potential sources of overfitting, the training dataset was augmented with random rotation, zoom, elastic distortion, and shear transformations to produce a total of 2,000 images (Figure 1d).
Network training halted after 37 epochs, once segmentation error (binary cross-entropy) on the validation dataset converged to a minimal value (Supp. Figure 1, Video 1). After hyperparameter tuning, the final model performance was assessed on a novel PDAC testing dataset, previously unseen by the network. Performance was quantified with the intersection-over-union (IOU) metric, which is defined as the overlap between the predicted and actual organoid pixels divided by their union (Figure 2a). An IOU greater than 0.5 is generally considered to reflect a good prediction and we chose this value to be the minimal benchmark for satisfactory model performance. All PDAC images passed the benchmark, with a mean IOU of 0.74 (SD = 0.081) (Figure 2b). Because the network was trained and assessed solely with images of PDAC organoids, we were also curious to evaluate its capacity to generalize to organoids derived from non-PDAC tissues. 18 microscopy images of organoids derived from lung epithelia, colon epithelia, and salivary adenoid cystic carcinoma (ACC) were manually segmented. The OrganoID network passed the benchmark for all non-PDAC images, with a mean IOU of 0.79 (SD=0.096). These results demonstrate the capability of the OrganoID neural network to reliably detect organoids from various tissues of origin in a pixel-by-pixel manner (Figure 2c).
The network was also evaluated for appropriate exclusion of non-organoid technical artifacts. Air bubbles in culture media or gel matrix were rarely detected by OrganoID with a false positive rate of 4.2% (Supp. Figure 2a). We observed that OrganoID segmentations also avoided cellular debris or dust embedded into the gel, ignored chamber borders, and performed robustly across microscope resolutions, organoid concentrations, and organoid shapes (Supp. Figure 2b-g). Each image was segmented in ∼300 milliseconds on a laptop CPU (Intel i7-9750H, 2.6GHz) with less than 200 megabytes of RAM usage.
Identification and tracking of individual organoids with diverse morphology and size
The convolutional neural network detects organoids in an image on a pixel-by-pixel basis, which can be used to measure bulk responses. For single-organoid analysis, pixels must be grouped together to identify individual organoids. This task is straightforward for isolated organoids, where all high-belief pixels in a cluster correspond to one organoid, but is more difficult for organoids that are in physical contact. To address this challenge, we developed an organoid separation pipeline (Figure 3a) that uses the raw network prediction map to group pixels into single-organoid clusters. Conventionally, neural network image segmentation methods set an absolute threshold on predicted pixels to produce a binary detection mask. This approach is effective, but discards useful information about the strength of predictions. The segmentations that were used to train the neural network were produced with a thin boundary between organoids in contact. As a result, the network predictions were marginally less confident about pixels near organoid boundaries. We took advantage of this phenomenon to identify and separate organoid contours with a modified Canny edge detector20 and a watershed transformation. Our detector sequentially applies (i) a pair of Sobel operators, which compute the image intensity gradient; (ii) a Gaussian filter, which smooths noisy regions; and (iii) a hysteresis-based threshold, which identifies locally strong edges. Edges are removed from the thresholded prediction image to mark the centers of each organoid. These centers are then used as initializer basins for a watershed transformation applied to the raw prediction image. The image is further refined to remove organoids that may be partially out of the field-of-view or are below a particular size threshold. The pipeline outputs a labeled image, where the pixels that represent an individual organoid are all set to a unique organoid ID number, which can be used for single-organoid analysis (Figure 3a).
For quantitative validation, OrganoID was used to count and measure the area of organoids in images from the PDAC and non-PDAC testing datasets (a total of 28 images). These data were then compared to the number and area of organoids in the corresponding manually segmented images. Organoid counts agreed with a concordance correlation coefficient (CCC) of 0.96 [95% CI 0.91-0.98]. OrganoID, on average, detected 0.5 fewer organoids per image than manual segmentation. The limits of agreement between OrganoID and manual counts were between −8.5 and 7.5 organoids (Figure 3b). Organoid area comparison demonstrated a CCC of 0.95 [95% CI 0.94-0.96]. OrganoID area measurements were biased to be 0.91×103 μm2 larger per organoid with limits of agreement between −5.47×103 μm2 and 7.28×103 μm2 (Figure 3c). Overall, the measurements produced by OrganoID were in considerable agreement with those obtained by hand, which supports the use of OrganoID for automated single-organoid analysis.
OrganoID was also developed with a tracking module for longitudinal single-organoid analysis of time-course imaging experiments, where changes in various properties, such as size and shape, of individual organoids can be measured and followed over time. The central challenge for the tracking module is to match a detected organoid in a given image to the same organoid in a later image. The Hungarian assignment algorithm21 was used to minimize a cost matrix based on the number of shared pixels between detected organoids in two images. This approach produces organoid “tracks” for unique organoids in time-lapse images. For validation, microscopy images taken every 4 hours from a 92-hour organoid culture experiment were passed through the entire OrganoID pipeline to produce growth curves that followed single-organoid changes over time (Figure 3d, e). The tracking step was also performed by hand to evaluate automated performance, which maintained over 89% accuracy throughout the duration of the experiment (Supp. Figure 3, Video 2).
OrganoID measures bulk and single-organoid death responses over time
PDAC organoids were treated with serial dilutions of gemcitabine (3nM-1000nM), an FDA-approved chemotherapeutic agent commonly used to treat pancreatic cancer, and imaged every 4 hours of the course of 72 hours. Propidium iodide (PI), a fluorescent reporter of cellular necrosis, was also added to the culture media to monitor death responses. Brightfield images were then processed with the OrganoID platform to identify organoids and analyze bulk and single-organoid responses (Figure 4a). At gemcitabine concentrations above 3 nM, the total organoid area increased for the first several hours, which reflected initial organoid growth, but then decreased to a value and at a rate inversely proportional to gemcitabine concentration (Figure 4b). Identified organoid counts for gemcitabine concentrations above 10 nM also sharply decreased over time. The total fluorescence intensity of the PI signal increased over time to a value and at a rate proportional to gemcitabine concentration, however the response to 100 nM gemcitabine appeared to induce a stronger death signal than the response to 1000 nM gemcitabine (Figure 4c). Measurements of death stains such as PI are typically normalized to a viability measurement that accounts for differences in the number and size of organoids between replicates that can compound over the duration of an experiment. We used OrganoID to normalize fluorescence measurements: the total fluorescence intensity of identified organoid regions was divided by the total organoid area at each time point. Normalization with OrganoID increased the separation of responses between each treatment group over time, decreased standard error across replicates, and corrected the response discrepancy between the 100 nM and 1000 nM conditions (Figure 4c). We also compared these normalized measurements to normalization with an MTS proliferation assay, which can only be used for endpoint analysis. Normalization with OrganoID identified the same significantly different endpoint responses as MTS assay normalization (Supp. Figure 4, Supp. Table 1).
Changes in organoid morphology can indicate important phenotypic responses and state transitions30. OrganoID was used to identify and profile single organoids in images at the experiment endpoint for gemcitabine concentrations of 100 nM, 10 nM, and the negative control. The fluorescence of individual organoids did not significantly differ across concentrations of gemcitabine (Supp. Figure 5), however fluorescence normalized to single-organoid area was significantly different (p<0.001). Organoid circularity, a morphological measurement that compares the organoid perimeter to the perimeter of a perfect circle, was also significantly different across gemcitabine dosages (p<0.001), which supports disruption of organoid architecture by gemcitabine (Figure 4d). This worked example demonstrates the advantages of OrganoID for automated bulk and single-organoid morphological analysis of time-course experiments without the need for live-cell fluorescence techniques.
DISCUSSION
Organoids have revolutionized biomedical research through improved model representation of native tissues and organ systems. However, the field has yet to fully enter the high-throughput experimental space. A central bottleneck is the challenge of automated response measurement and analysis in large numbers of microscopy images. Organoids exhibit striking diversity in morphology and size and can move through their 3D environment into and out of the focal plane; current image processing tools have not quite been able to capture these aspects in a robust manner. We developed OrganoID to bridge this gap and automate the process of accurate pixel-by-pixel organoid identification and tracking.
Experimental replicates in organoid studies can differ in the number and distribution of sizes of organoids. This difference between identical conditions requires per-sample normalization of response measurements to account for differences in baseline growth of organoid colonies. There are several commercially available live-cell assays that can facilitate normalization in 2D culture. However, these same assays have proven to be difficult for organoid use due to the production of toxic photobleaching byproducts, limited diffusion through the gel matrix, and nonspecific staining of the gel matrix that results in a considerable background signal. Another available option is to genetically modify each organoid sample to express fluorescent proteins, however this increases experimental time and complexity and may alter cellular dynamics from the original tissue sample. The OrganoID platform can be leveraged for accurate normalization of standard organoid assays without live-cell fluorescence methods. OrganoID is also uniquely useful for efficient quantification of single-organoid morphological features, such as circularity, that can reflect important dynamic responses.
Additionally, we have contributed a manually segmented organoid image dataset for use in other computational platforms. OrganoID has demonstrated compatibility with organoids of various sizes, shapes, and sample concentrations as well as various optical configurations. Most excitingly, the OrganoID model was trained and validated on images of PDAC organoids but still demonstrated excellent generalization to images of other types of organoids, including those derived from colon tissue, lung tissue, and adenoid cystic carcinoma.
OrganoID was trained with and tested on a diverse, but relatively small set of images. Despite the suggested generalizability of OrganoID to various samples and optical configurations, performance may still differ with other types of organoids or microscopes. As well, OrganoID can only detect and assign a single organoid to each pixel in an image. While the platform can appropriately identify contours of organoids in physical contact, it cannot distinguish organoids that overlap across the focal plane. These limitations can be overcome with additional validation, and perhaps an expanded training dataset, as well as the use of multiple focal planes for image analysis.
We have released the OrganoID platform open-source and freely licensed on GitHub. The repository includes all source code and a compiled executable, as well as the entire training and testing dataset, usage instructions, and scripts used for the examples presented in this paper. The network training module is also included on the repository to allow further model training to improve performance for any untested applications.
Author contributions
J.M.M., B.S., S.S.K. and S.T. were involved with the conception of the platform. J.M.M. and B.S. wrote the manuscript, collaborated on platform design, and created the manual organoid segmentation dataset. B.S. performed all organoid experiments and the associated data/statistical analysis and interpretation. S.S.K developed and evaluated the initial deep learning segmentation approach. J.M.M. optimized the neural network, developed the single-organoid pipeline, and implemented the OrganoID software architecture, image processing algorithms, and performance quantification tools. A.R., M.B., S.T. supervised the work and provided technical support. All authors commented on the manuscript.
Competing Interests
The authors declare no competing interests.
Correspondence
Savaş Tay, Institute for Molecular Engineering, The University of Chicago, tays{at}uchicago.edu
Funding
This work is supported by NIH R01 GM127527 and P. G. Allen Distinguished Investigator Award to S.T.
METHODS
Organoid culture and image acquisition
Tumor organoid cultures derived from pancreatic adenocarcinoma (PDAC) patients were isolated and prepared as previously described22. PDAC organoids were cultured and imaged in a microfluidic platform19 or a 24-well suspension culture plate (Thermo Fisher, 144530). Human colonic organoids were cultured similarly to PDAC organoids, with Matrigel and growth media as described in established methods23. Distal respiratory organoids were obtained from the Ishay-Ronen Lab at the Sheba Medical Center and cultured through a previously described protocol24 in 24-well plates. Images of adenoid cystic carcinoma (ACC) organoids were directly obtained from Dr. Weber and colleagues from University of Chicago Medicine. The pancreatic and airway lung organoids were cultured and imaged on an automated translational stage of an inverted microscope (Nikon Eclipse Ti) enclosed in an environmentally controlled chamber (Life Imaging Service GmbH, Basel, Switzerland). The enclosure provides temperature, humidity, and CO2 gas control to maintain adequate cell culture conditions for the organoids. Organoids were cultured at a constant 37°C, 5% CO2, a humidity flow rate of 25-30 L/hour, and 95-100% relative humidity. Images of the organoids were acquired through the standard microscope software that is capable of automatically acquiring images at different positions, Z-planes/stacks, and multiple fluorescent filters (NIS-Elements software, Japan). The microscope was equipped with a digital complementary metal-oxide semiconductor (CMOS) camera (ORCA-Flash 4.0, Hamamatsu, Japan), which imaged the organoids using a x10 objective at 2 to 4-hour intervals.
Neural network design and implementation
The neural network architecture was based on the u-net encoder-decoder network, which has produced successful results for a variety of biomedical image segmentation tasks9. u-net uses several convolutional filters for each layer of neurons and doubles the number of filters per-channel for each sequential layer. The original u-net implementation uses 64 filters in the first layer, which results in quite a large number of trainable parameters across the full network (over 30 million). We sequentially reduced the number of filters in the first layer by powers of two to reach a minimal value that preserved performance on the validation dataset. The final OrganoID u-net structure uses only 8 filters in the first layer, which results in a network structure with less than 500,000 trainable parameters (a 98% reduction compared to the original implementation). This simplification reduces the computational power needed to train and use the network and, in theory, decreases network overfitting. All convolutional neurons were set to compute outputs with the exponential linear unit activation function10. The final 1×1 convolution was set with a sigmoid activation function to produce a normalized output that corresponded to the probability of an organoid at each pixel. All images are auto-contrasted and resized to 512×512 pixels before training and inference. Python was used for the entire OrganoID platform and Keras (an interface to the TensorFlow library) was used for network expression, training, and operation. The TensorFlow Lite API was used to minimize the memory footprint and number of software dependencies required for network inference in the OrganoID standalone distribution.
Training procedure and ground-truth dataset creation
Ground-truth segmentations were created by hand with Pinta and Aseprite, two open-source image editing programs. 66 ground-truth segmentations were created from brightfield and phase-contrast microscopy images of PDAC organoids and were randomly spilt into datasets for training (52 image pairs) and validation (14 image pairs). The Augmentor Python package25 was used to apply random rotation, zoom, shear, elastic distortion, and skew operations to the training dataset for data augmentation. The neural network was trained with the Adam stochastic optimization algorithm26 at a learning rate of 0.001. Unweighted binary cross-entropy between predicted and ground-truth segmentations was used for the loss function. Layer weights were initialized with the He method27. An early stopping rule was used to halt training once performance on the validation dataset reached a minimum (i.e. once 10 epochs pass with no improvement in validation loss). The batch size was set to train on 8 augmented images for every round of backpropagation. Dropout regularization was introduced after all convolutions to randomly set 12.5% of neuron weights to zero after each batch. After each epoch, a copy of the model was saved for additional evaluation of the training process.
Dataset for evaluation of network performance and generalizability
The network structure and training process were manually tuned for performance on the PDAC validation dataset. After optimization, we locked out any further changes to all hyperparameters and evaluated final network performance and generalizability on a separate dataset of 28 organoid microscopy images. The dataset included images of organoids derived from PDAC (10), benign colon tumor cells (6), long epithelia (6), and salivary adenoid cystic carcinoma (6). The images were obtained through brightfield and phase-contrast microscopy from multiple microscopy cores and manually labeled by two independent reviewers.
Single-organoid identification and segmentation refinement
The scipy28 and scikit-image29 packages were used to identify individual organoids from the network detection images. Detection images were thresholded at 0.5 and passed through a morphological opening operation to remove weak and noisy predictions. Partial derivatives were computed with a Sobel filter, passed through a Gaussian filter (s=2), and converted to an edge mask with a hysteresis threshold filter (Thi=0.05, Tlo=0.005), which was then was used to mark organoid centers. The watershed method was used to identify filled organoid contours, with the organoid centers as label initializers and the network prediction image as an inverted heightmap. Labeled organoids were morphologically filled and discarded if the total area was below 200 pixels. Images were morphologically processed with scikit-image to record properties of identified organoids.
Tracking of individual organoids over time
The tracking algorithm builds a cost-assignment matrix for each image in a time-lapse sequence, where each row corresponds to an organoid from the previous image and each column corresponds to an organoid in the next image. Each matrix entry is the cost of assigning a given organoid detection in the current image to a detection in the previous image. The cost function was defined as the inverse of the number of shared pixels between the two organoids. As such, the assignment cost between two detections will be minimal for those that are of similar shape and location. The cost-assignment matrix is also padded with additional rows and columns to allow for “pseudo-assignments” that represent missing or newly detected organoids. Finally, the Munkres variant of the Hungarian algorithm21 was used to minimize the cost-assignment matrix to find an optimal matching between organoid detections in the previous image and the detections in the current image
Statistical methods for platform validation
Statistical analysis was performed with the numpy and scipy packages in Python. Network performance on the testing dataset was evaluated as the pixel-wise intersection-over-union (IOU) of predictions compared to ground-truth segmentations. A single IOU value was computed for each prediction/ground-truth pair and summarized as a mean IOU and standard deviation over the dataset. For agreement of single-organoid counts and measurements, the Lin concordance correlation coefficient (CCC) was computed as The Fisher transformation (artanh(CCC)) and the inverse Fisher transformation (tanh(zLO, ZHI)) were also used to obtain a z-value and construct a 95% confidence interval for CCC statistics. All plots were generated with the matplotlib and seaborn packages in Python.
Drug screening experiments
PDAC organoids were grown for one week in standard culture conditions and then treated with gemcitabine hydrochloride (G6423, Sigma) at six serial dilutions from 3 nM to 1000 nM with negative control. Propidium iodide (Thermo Fisher P3566) was used to fluorescently measure cellular death and relative viability of organoids in real-time at 4-hour intervals over 72 hours. An MTS proliferation assay (Promega, G3580) was also performed at the end of the experiment to determine cell viability for each condition according to manufacturer’s instructions. Statistical hypothesis testing was performed with scipy and MATLAB. Organoid circularity was computed as the circumference of a circle with equivalent area divided by the actual perimeter of the organoid.
Validation rigor and indications of robustness
The OrganoID neural network was tested on images of four types of organoids taken at multiple microscopy cores. Images were manually segmented by two independent evaluators. OrganoID single-organoid count and area measurements were compared against the corresponding measurements of the manual test segmentations. OrganoID tracking results were compared to manually tracked images. The OrganoID platform and test microscopy images were distributed to an independent user to confirm reproducibility of all metrics, statistics, and figures.
Acknowledgements
We would like to acknowledge Dr. Dana Ishay-Ronen and Dr. Rakefet Ruth Ben-Yishay from Sheba Medical Center for providing the airway lung organoids; Dr. Sonia Kupfer and Margaret Bielski from University of Chicago Medicine for providing the colon organoids; Dr. Chris Weber and Dr. Le Shen from University of Chicago Medicine for providing the adenoid cystic carcinoma (ACC) organoids.