ABSTRACT
Deep learning methods have outperformed previous techniques in most computer vision tasks, including image-based plant phenotyping. However, massive data collection of root traits and the development of associated artificial intelligence approaches have been hampered by the inaccessibility of the rhizosphere. Here we present ChronoRoot, a system which combines 3D printed open-hardware with deep segmentation networks for high temporal resolution phenotyping of plant roots in agarized medium. We developed a novel deep learning based root extraction method which leverages the latest advances in convolutional neural networks for image segmentation, and incorporates temporal consistency into the root system architecture reconstruction process. Automatic extraction of phenotypic parameters from sequences of images allowed a comprehensive characterization of the root system growth dynamics. Furthermore, novel time-associated parameters emerged from the analysis of spectral features derived from temporal signals. Altogether, our work shows that the combination of machine intelligence methods and a 3D-printed device expands the possibilities of root high-throughput phenotyping for genetics and natural variation studies as well as the screening of clock-related mutants, revealing novel root traits.
Introduction
Plants are sessile organisms unable to seek out optimal environmental conditions for development and survival. Strikingly, a remarkable developmental plasticity allows plants to complete their life cycle under changing growth conditions1. Understanding plant root plastic growth is crucial to assess how different populations may respond to the same soil properties or environmental conditions and to link this developmental adaptation to their genetic background2. Under controlled conditions, root development is generally observed based on images of plants growing vertically on the surface of a semisolid agarized medium. Root system architecture (RSA) is then characterized by parametrization of a grown plant, which relies on the combination of a subset of variables like main root (MR) length or density and length of the lateral roots (LRs)3. Several semi-automatic tools have been developed to assist root phenotyping at specific time points4. However, temporal phenotyping is generally hindered by technological limitations, ignoring potentially useful phenotypical parameters that may be linked to the temporal dynamics of root growth. Here we present ChronoRoot, a low-cost system based on off-the-shelf electronics, 3D printed hardware components and deep learning models, allowing high-throughput temporal phenotyping of Arabidopsis thaliana RSA. Figure 1 illustrates the different components of ChronoRoot. Temporal sequences of pictures, automatically snapped, are processed for root segmentation through a new convolutional neural network (CNN) architecture. We leverage the latest CNN architectures for image segmentation and propose a CNN architecture for RSA delineation which incorporates deep supervision, producing fast and accurate segmentations. The root extraction workflow is completed by a temporal consistency refinement step and a final graph generation process, which generates a labeled root graph for every image. An exploratory approach assessing root growth under alternative photoperiods served to demonstrate that temporal phenotyping performed by ChronoRoot allows deciphering the evolution of the traditional RSA parameters throughout time. Moreover, novel parameters emerged, including architectural features, oscillating growth speed and other characteristics derived from spectral analysis of the growth signals in the Fourier domain. The combination between a low-cost automatic device for image acquisition and machine intelligence methods for image segmentation gave rise to a powerful tool for root phenomics potentially applicable to natural variation studies, the characterization of root-related subtle disorders and the screening for clock-associated mutants.
Results
We designed an automatic method to perform RSA delineation in temporal image sequences of plant roots. Our model takes a sequence of images as input and outputs a labeled graph for each frame, representing the current root growing state. Graphs are powerful data structures particularly useful to represent curvilinear shapes like plant roots (details on the graph generation process are provided in the Methods section). The main module of the RSA delineation method is a deep convolutional neural network which produces a dense segmentation mask, where every pixel is classified as belonging to the root or the background. We proposed different CNN architectures for this task (described in the Methods section), and compared their performance with state-of-the-art models using manual annotations produced by expert biologists. We measured three different metrics: (1) Dice coefficient quantifies the overlapping between the prediction and the ground-truth, (2) Haussdorf distance indicates the maximum distance between them and (3) the recall (or sensitivity) refers to the fraction of root pixels retrieved over the total amount of root pixels. Quantitative results are included in Table 1. Based on these results, we chose two models, depending on whether we aim at having a faster or more accurate method:
Fast method: The fastest models are the proposed UNet variants, requiring up to half a second to process a high resolution image using a standard GPU. These models have lower parameter complexity compared to state-of-the-art architectures like SegNet and DeepLab, which explains the lower running time. Among the fast UNet models, we observed that the proposed Deeply Supervised ResUNet (DSResUNet) shows a significantly lower value for Hausdorff distance, while keeping equivalently good Dice and Recall. The proposed DSResUNet architecture (depicted in Figure 1) combines residual blocks5 with deep supervision6, improving the results of a standard UNet with a minimum increase in model complexity.
Accurate method: We proposed to combine all the implemented architectures into a single ensemble method, increasing model diversity by creating an ensemble of multiple models and architectures7. This ensemble of deep models increased the running time by a factor of 9, but achieved the best performance across all metrics, outperforming state-of-the-art models like SegNet and DeepLab.
ChronoRoot implements both fast and accurate variants, giving the users the opportunity to decide according to their requirements. In this study, we used the fast method based on the proposed Deeply Supervised ResUNet model, which offered a good trade off between running time and accuracy. As shown in Figure 1, we apply several post-processing steps after segmentation, which are independent of the CNN model. We first apply a Conditional Random Field (CRF)8, 9 model to improve the homogeneity of the labels assigned to neighboring pixels. Then, we enhance the temporal consistency of the segmentations by considering its weighted average. These steps serve to remove spurious segmentations by analyzing a temporal sequence of images, which ultimately translates into generating more stable phenotypic measurements. A graph structure is then constructed where every node is assigned a class label indicating whether it is associated with the plant seed, main root, lateral root, bifurcation or root tip. Temporal consistency on the graph structures is finally improved by tracking the labeled nodes and solving conflicting cases. A more detailed description of these steps can be found in the Methods section. After the graph generation process, we proceed to extract phenotypic features for RSA characterization.
Temporal dimension of traditional and novel RSA parameters
We analyzed temporal sequences of plant roots growing under different conditions. In order to assess the potential of ChronoRoot, we decided to compare RSA of Arabidopsis thaliana ecotype Col-0 grown under two distinctive photoperiods, i.e. long day (LD; 16 h of light, 8 h of dark) or continuous light (CL; 24 h of light). Light availability and photosynthesis in the shoot determine the amount of sugar transported to the roots, thus modulating underground plant growth. Moreover, ample evidence suggests that root developmental plasticity depends on the light environment, involving a more sophisticated impact on endogenous signaling pathways10.
Traditional parameterization of RSA expanded to temporal dynamics revealed the progression of root growth under continuous light (CL) and long day (LD) conditions. A representation of root automatic segmentation is shown in Figure 1. Our experiments show that main root (MR) length, the sum of lateral root (LR) length and the resulting total root system (TR) begin to differ between conditions at approximately 200-250 h (8-10 days) after germination (Figure 2A, B and C), together with LR number (Figure 2D). Notably, root growth was not only faster under CL, but also resulted in a different RSA, exhibiting a higher density of LRs and a lower component of the MR over the total root system (Figure 2E and F). Notably, between 250 h (10 days) and the end of the experiment (336 h, 14 days), the contrast between both photoperiods increased gradually in every measured parameter, hinting at a temporal reorganization of root development under different light conditions.
Based on the information derived from temporal phenotyping, we explored in more detail the reconfiguration of RSA under alternative photoperiods. We identified the time point at which the sum of LRs length equals the MR length as a novel parameter of RSA dynamics (Figure 3A). However, no significant difference was observed in the distribution of individual time-length points of plants grown under CL or under LD (Figure 3B). The analysis of the relationship between the MR and LRs along time, determined by the difference between both measurements (MR-LRs) aligned to the time point at which MR and LRs are of the same length (time 0) is shown in Figure 3C. It reflects that the difference between MR and LRs tends to have a significantly larger absolute value for plants grown under CL than under LD. Moreover, we extracted different indicators to analyze the dynamics of these curves. Figure 3D shows the approximate derivative (computed by means of finite differences) at the time point at which MR and LRs are of the same lengths (time 0). These differences are not statistically significant according to a Mann Withney U test. However, when extending the analysis to the full +/−24hs range by fitting a linear function to every curve from Figure 3C and plotting the corresponding slopes (Figure 3E), we found strong differences in the distribution (statistically significant according to a Mann Withney U test) for both photoperiods.
In order to assess the impact of RSA reconfiguration on the area explored by roots under distinct photoperiods, we calculated the dynamic convex hull for each subset of plants. Interestingly, the observation of the convex hull resulting from the overlap of all individuals grown in the corresponding conditions reveals an extended high density of LRs along the MR axe at the end of the experiment under CL (14 days after germination, Figure 4A, B and C). Notably, the area of the average convex hulls between 8 and 14 days does not differ between CL and LD conditions (Figure 4D). Nonetheless, the quantification of the sum of LRs length over the convex hull area indicates that the density of LRs is higher under CL between 10 and 14 days (Figure 4E). Collectively, our analyses indicate that global LR length increases under CL as a result of more numerous LRs growing simultaneously, although the area explored by the RSA does not differ between the two photoperiods. Thus, the global density of the resulting RSA is higher under CL.
Novel speed-based parameters derived from temporal phenotyping
The information derived from the temporal dimension of traditional and novel RSA parameters indicated that the difference in root growth rate became broader throughout time under CL vs. LD. It has been shown that the Arabidopsis MR exhibits an oscillating growth which likely depends on the lunisolar tide10, 11 and light-associated carbon partitioning12. Therefore, based on the segmentations obtained with our deep learning models, we calculated the growth speed throughout the experiment in both conditions, showcasing how novel speed-based parameters can be derived via ChronoRoot. MR speed grew steadily until approximately 150 h under LD and 200 h under CL post germination, and the average maximum speed reached in CL was higher than in LD (Figure 5A). Strikingly, the difference in the growth speed of the global root system (TR) between the two conditions became increasingly larger since the moment when the speed of the MR was stabilized (Figure 5B), hinting at a different acceleration rate between conditions. The observed root growth dynamics further supports the rising relevance of LRs as a main component of RSA throughout time. Notably, the analyses of growth speed uncovered an oscillating behavior in both conditions (Figure 5A and B). A Fast Fourier Transformation (FFT) of the signal of MR growth speed in CL vs. LD revealed a major energy difference in the components corresponding to the frequencies of 1/24h and 1/12h, respectively (Figure 5C). Remarkably, these two components distinguish circadian and ultradian rhythms displayed by plants grown in LD, with a pronounced local minimum of the growth speed at 1/24h and a minor local minimum at 1/12h (Figure 5D). Strikingly, the most pronounced differences revealed by FFT (Figure 5C) served to uncover a root growth clock-related disorder suffered under CL, coinciding with a blurred daily oscillation of growth speed, in comparison with the corresponding sine curves (Figure 5D; a detailed comparison of MR, LR and TR growth speed analyses is shown Supplementary Figure 1). Although an oscillating behavior can be observed under CL towards the end of the experiment (Figure 5A and B), the energy at 1/24 and 1/12 frequencies was higher under a LD photoperiod throughout the complete time lapse analyzed (Figure 5E and F). Notably, the difference between conditions of the TR oscillating speed of growth is mainly due to the MR contribution (Supplementary Figure 2). Altogether, our study of wild-type Arabidopsis Col-0 plants growing under alternative photoperiods using ChronoRoot served to reveal novel temporal parameters of root development, notably including clock-related features depending on the light environment.
3D-printed device for temporal image acquisition
The ChronoRoot device is an affordable and modular imaging system based on 3D-printed and laser cut pieces and off-the-shelf electronics (Figure 1.1 and 1.2). Each module consists of a Raspberry Pi (v3)-embedded computer controlling four fixed-zoom and fixed-focus cameras (RaspiCam v2), and an array of infrared (IR) LED back-light. In between each camera and the corresponding IR array, there is a vertical 12 × 12 cm plate for seedling growth, allowing automatic image acquisition repeatedly along the experiment without any modification or movement of the imaging setup. The four-plate module is small (62 × 36 × 20 cm) and can be placed in any growth chamber. The different parts of the imaging setup (back-light, plate support and camera) can be positioned along a horizontal double-rail to control the field of view of the camera and accurate lightning. In addition, the camera can be moved vertically. ChronoRoot allows image acquisition at a high temporal resolution (a set of pictures every minute). The use of an IR back-light (850 nm) and optional long pass IR filters (> 830 nm) allow acquiring images of the same quality independently from the light conditions required for the experiment, during day and night.
Each module is connected to the network either by Wi-Fi or Ethernet cable. A web interface allows the control of the device offering live feed of the cameras for field of view and focus setup. The user can program the activation of cameras and IR back-light, starting and ending dates, the time basis for picture acquisition, and finally follow the progression of the experiment. The pictures are saved directly on an external drive plugged on the Raspberry Pi. Once the experimental setup is ready, each module is completely independent from the external environment and the access to the network (for more details see Methods, Supplementary Figures 3–7 and the Supplementary 3D printing and laser cutting files).
Discussion
The plant phenotype can be defined as the integration of structural, physiological, and performance-related traits of a genotype in a given environment. Plant phenotyping is therefore the act of determining the quantitative or qualitative values of these traits13. The advent of novel imaging technologies and image processing have revolutionized plant phenotyping, expanding the frontiers of phenotypic trait measurement. Plant roots have a major role in plant anchorage and resource acquisition while offering environmental benefits such as carbon sequestration and soil erosion mitigation14. The growing knowledge linking genetics with functional properties of plant roots is of crucial interest to plant breeding, notably for the design of novel strategies for sustainable agriculture and environmental stewardship in the face of the impending climate change. Whereas high throughput genotyping, sequencing-based genotyping and genomic breeding are behind current agricultural practices in the era of omics technologies, the collection of phenotypic data for a thorough characterization of the RSA is increasingly becoming a limiting factor15. Although significant advancements in the application of imaging sensors for high-throughput data collection have allowed comprehensive plant phenotyping16, the characterization of root traits has been hampered by the inaccessibility of the rhizosphere.
Large and sophisticated phenotyping platforms are deployed worldwide and allow the simultaneous phenotyping of several hundreds of plants (see the International Plant Phenotyping Network i). However, their big dimensions and high automatization reserve their implementation on specialized locations and mainly for large phenotyping experiments. In parallel, the advancement of the Do It Yourself (DIY) movement has promoted the development of a growing number of low-cost phenotyping devices combining 3D-printed, laser cut, captor and microcontroller coming from open-source and open-hardware communities such as Arduinoii or Raspberry Piiii. Successful inexpensive devices have allowed monitoring plant leaf area17, 18, RSA on soil19 or rhizotron20.
In the last few years, images of the root system from different plant species have been acquired manually using a flat-bed scanner or a camera positioned in front of vertical petri dishes. Root phenotyping is generally performed upon single-time-point images or using several images acquired during growth in time-lapse sequences. More recently, semi-automatic devices and softwares have also helped to increase the efficiency of image acquisition and associated analyses21–26. The great need of throughput in screening experiments to uncover the genetic basis of root development, justifies the use of simplified artificial culture conditions and standardized environments to make the RSA accessible to image acquisition13. Software tools use input images of root systems grown under a variety of conditions, including hydroponic and aeroponic systems, agarized medium, paper pouches or soil24, 25. Here we propose to use vertical square petri dishes for plant growth on the surface of transparent agarized medium, for automatic acquisition of photographs allowing a high resolution temporal phenotyping of the RSA.
According to Quantitative Plant27, 28, over 40 image processing softwares are available for root system analysis25, 29. RSA parameters are extracted from various types of 2D images captured from agar plates or washed roots extracted from soil. Moreover, 3D RSA reconstruction is possible using X-ray computed tomography30 or magnetic resonance imaging31. Nearly all the reported tools need human input to be operated and retrieve precise numbers, limiting the throughput of the experiment and hindering the temporal analysis of RSA parameters. Advances in machine learning applied to image analysis allowed partially overcoming these limitations. For example, CNN-based RootNav 232 is able to retrieve the root architecture from 2D grown plants with minor user intervention. Temporal root phenotyping is also possible using automatic tools which are limited to MR growth monitoring (e.g. RootTrace33 or BRAT34). Here we present ChronoRoot, a deep learning based model which employs CNNs and temporal monitoring of root growth for a comprehensive RSA dynamic characterization. ChronoRoot expands the possibilities for high throughput root phenotyping, which is of major importance for natural variation and GWAS, as well as mutant characterization and screening. Notably, it has been shown that clock-related mutants exhibit a differential oscillating MR growth under alternative conditions11, 12, 35. ChronoRoot offers an ideal platform for the identification of genotypes associated with altered clock traits, based on the analysis of spectral features extracted from temporal signals.
Methods
Plant material and growth conditions
Arabidopsis thaliana ecotype Col-0 seeds were surface sterilized and stratified at 4°C for 2d before being grown under long day conditions (16h light, 140μEm−2s−1/ 8h dark), or continuous light (24h light, 140μEm−2s−1) at 22°C, on half-strength Murashige and Skoog media (1/2 MS) (Duchefa, Netherlands) with 0.8% plant agar (Duchefa, Netherlands). Four seeds were used per plate.
Hardware description
An automated imaging setup was designed and built in the shape of an independent module of 62 × 36 × 20 cm (Figure 1). It is aimed at imaging up to four vertical plates either in color or in near-infrared (NIR) lightning. Each module consists of a single board computer (Raspberry Pi) controlling four cameras through a multiplexer module and an array of NIR LED illumination through a relay. The main support of each module is a 620 × 36 × 5 mm acrylic sheet cut using a laser cutter to allow to screw the different parts or let pass strips connecting the camera to the camera multiplexer. Several 3D pieces were designed and printed to place the different components of the module. Each module is separated in four subparts, each of them along a double aluminum axis. This axis allows to adjust the distance of the different parts of the imaging setup: NIR illumination, plate support, camera. The underpart of the module was used to fix the LED AC/DC adaptor, the relay and the computer. The supports under the platform raise and stabilize the module. The Annex includes a full description of the components and the steps for the assembly of the device. The Supplementary File includes the 3D-printing and laser cut plans.
Datasets
We generated two different datasets in this work: the first one was used to train and evaluate the segmentation performance of the CNN models, while the second one served as an exploratory use case, to assess root growth under alternative photoperiods and provide an example of the novel temporal phenotypical parameters that can be extracted with ChronoRoot. Note that all these images were obtained with the low-cost hardware previously described, and both are available to encourage reproducible research.
Dataset used to train and validate the deep learning models for root segmentation: The dataset used for training consisted of 331 images from 55 videos (on average 6 images from the same plate at different states of growth), 11 of those were annotated by an expert biologist. The dataset used for testing consisted of 55 images from 11 different videos, all annotated by the same expert. The tool used for the manual annotation was ITK-SNAP36.
Use case dataset for plant phenotyping under alternative photoperiods: We used 12 videos for each photoperiod, with pictures taken every 15 minutes. We took the first 17 days (1632 frames), and after processing the videos we proceeded to discard the results from the first 3 days prior to seed germination.
Computational methods
We evaluated different state-of-the-art architectures for image segmentation, and proposed new variants which achieved a good compromise between processing time, model complexity and accuracy, as discussed in the Results section. This segmentation module is followed by several post-processing stages including a CRF post-processing to enhance label homogenity, a temporal consistency refinement step, skeletonization, graph construction and node tracking. ChronoRoot outputs a labeled graph per image indicating which nodes correspond to the seed, main root, lateral roots, bifurcations and the root tips. For each time step, the complete RSA is saved following the RSML format37.
Deep learning models for root segmentation
CNNs are representation learning methods with multiple abstraction levels, which compose simple but nonlinear modules transforming representations at one level into a representation at a higher, slightly more abstract level38. These models are specially suited for computer vision tasks, in particular for image segmentation39. We explored six different convolutional neural network architectures to perform plant root segmentation. Four of them are state-of-the art existing architectures, while the other two were proposed in this work. In what follows, we first present a brief description of the state-of-the-art architectures (namely the UNet40, ResUNet41, SegNet42 and DeepLab43) and then discuss the two models proposed in this work.
U-Net
The first model is a modified lightweight version of the standard U-Net40, which employs a fully convolutional encoder-decoder architecture and produces a dense segmentation map at the pixel level. Based on the original UNet model, we implemented a lightweight version reducing by 4 the number of feature maps per convolutional layer. Skip connections were implemented via summations of the signals in the up-sampling part of the network, instead of the concatenation used in the original version. We also replaced the max-pooling layers with avg-pooling, and used ELU as non-linearity instead of RELU. See Supplementary Table 1 for a detailed description of the implemented architecture.
ResUNet
For the second model (ResUNet), we replaced the convolutional layers in the aforementioned U-Net architecture by residual blocks5. Residual blocks help to prevent the degradation problem which occurs in very deep neural networks by learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. Recent works suggest that residual blocks are effective at segmenting tubular structures like plant roots or roads in a map41. See Supplementary Table 2 for a detailed description of the implemented architecture.
SegNet
The SegNet architecture42 is a fully convolutional encoder-decoder neural network, widely adopted by the computer vision community to perform dense image segmentation. The architecture of the encoder is identical to the first 13 layers of VGG-1644 and the role of the decoder network is to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise classification. Differently from the U-Net where skip connections are used to propagate the complete feature maps from the encoder to the decoder, the upsampling in the decoder part of the SegNet model uses the memorized max-pooling indices from the corresponding encoder level. Our implementation was based on a publicly available modeliv.
DeepLab v3
The DeepLab V3 model43 follows a different approach to generate dense segmentation maps. Differently from the previous models which use skip connections (UNet) or memorized max-pooling indices (SegNet), this model employs atrous convolutions with upsampled filters to extract dense feature maps and capture long range context.
Proposed Models
On top of these state-of-the-art architectures, we propose two different CNN models. In the first model, named DSResUNet, we aimed at improving the segmentation accuracy while keeping at the same time a fast lightweight model. In the second case, we focused on increasing the robustness and boosting the accuracy of the segmentation method, at the expense of a more complex model which follows the principle of ensemble learning.
DSResUNet
Taking the ResUNet as a baseline model, we propose here a new architecture which combines residual connections and deep supervision6 to improve the accuracy of the results. Deep supervision integrates additional loss terms which are computed using feature maps from the intermediate CNN layers, instead of the last one only. We concatenated the ResUNet output with the original input image, and processed these feature maps with two additional convolutional layers. This resulted in a cascade of two networks which are trained jointly, where the first one produces an initial segmentation map that is then refined by the second part of the network. We computed two loss terms, one after the output of the standard ResUNet and another one after the additional convolutions. The sum of both terms constitute the loss function used to train the DSResUNet model. See Figure 1 for a graphical illustration of the architecture, and Supplementary Table 2 for a detailed description.
Ensemble
Our final segmentation method is an ensemble model. The idea of ensembling is that we can create higher performing models by combining multiple predictors using an aggregation function. One of the most common strategies to implement ensemble models is bagging45, where the same classifier is trained multiple times using different samples of the training set, and the final output is obtained as the average of the independent predictions. In this work, we followed a different principle which had been successfully applied in the context of medical image segmentation, where instead of combining several instances of the same model trained with different training samples, we combined different models and architectures trained with the same datasets7. The idea is to average out the bias infused by individual model configurations, to approximate more reliably the true posterior distribution. In the context of image segmentation, given a dataset L = (x, y)i where x is an intensity image and y the corresponding ground truth segmentation, we aim at learning the underlying conditional distribution P(y|x) which maps input images x into segmentation maps y. This is commonly approximated by a model P(y|x; θm) which has trainable parameters, determined in our case by the neural network architecture. These parameters were learnt so that they minimize a particular loss function (see next section for more details in the loss functions used in our work) using the dataset. Given different architectures (in our case,), we obtained independent estimates of and combined them following7 approximating the posterior P(y|x) as:
We implemented this ensemble of multiple models and architectures by averaging the predictions of the 5 previous models (UNet, ResUNet, DSResUNet, SegNet and DeepLab v3), obtaining a more robust and accurate segmentation method that significantly outperforms the independent instances.
Training details
All the CNN models were trained using binary cross entropy as the loss function, Adam optimizer with default parameters, learning rate of 0.0001 and weight decay = 1e-8 for UNet-like models, 1e-9 for DeepLab and 1e-10 for SegNet. The hyper-parameters were chosen by grid search using the validation data. All models were implemented in TensorFlow 1, and the source code is publicly availablev. The training was done on a standard workstation with Intel(R) Core(TM) i7-8700 CPU, 64 GB RAM and a NVIDIA Titan × graphics processing unit.
Since we are dealing with a relatively small dataset, data augmentation was crucial to achieve good segmentation performance. We implemented online data augmentation through a variety of patch-based augmentation procedures including addition of Gaussian noise, random Gamma corrections to simulate different lighting conditions, artificial blur and horizontal flipping. These transformations were applied to both the images and their corresponding ground-truth segmentation masks. The proposed architectures are all fully convolutional, enabling a patch-based training procedure. As this is a highly unbalanced problem (we have fewer pixels corresponding to root class than background) we implemented the following patch sampling strategy: we sampled patches from random positions centered in root pixels with the same probability as patches centered in background pixels. After performing a grid search of hyperparameters, the size of the training patches was set to 256×256 and we used batches of 8 patches. At test time, we worked with the full resolution images which can be fed to the network and processed by the fully convolutional architectures.
CRF Post-processing
The CNN segmentations are post-processed using a standard fully connected conditional random field (CRF)46. The CRF operates under the hypothesis that pixels which are contiguous and have similar intensity values should be assigned the same label. We used an efficient publicly availablevi implementation of a dense CRF9 publicly available with Potts compatibility function and hand-tuned parameters θ = 5 and θ = 3.
Graph generation and temporal consistency improvement
The CNN output can be interpreted as a soft segmentation. Since we processed temporal sequences of growing plant roots, we applied a post-processing step to improve temporal consistency that consists in taking the average of the probability maps of a given frame and the previous one, iteratively, using the new value for the next step. This allowed us to get rid of spurious segmentations generated by the CNN models in the background, enhancing the quality of the segmentations along the root. The average helped to alleviate certain problems caused by root occlusion, as the probability maps associated to previous frames act as memory mechanisms, resulting in more stable segmentations with little to zero noise caused by box borders or occlusions from water droplets.
At this point, as the user selects a Region of Interest (ROI) for each plant, the algorithm starts working one by one. We proceed to threshold the accumulated probability map for the selected plant, perform closing and opening morphological operations47 to eliminate spurious pixels and then we select the biggest connected component as the root segmentation. Finally, we proceed to skeletonize48 the segmentation and construct a graph that represents the root system architecture.
We run a deep first search (DFS) algorithm49 in order to label the bifurcation and end nodes of the root graph. Labels were assigned for the seed, main root tip, bifurcation and lateral root tip nodes. Graph matching was performed between the labeled nodes of successive graphs in the temporal sequence to track the evolution of the root. These graph structures allowed us to extract phenotyping features such as main root length, total lateral roots length or number of lateral roots at every temporal step. By processing the complete temporal sequence for a given root, we can obtain temporal features such as growing speed or information about the root behavior on day-night cycles, enabling the emergence of novel temporal plant phenotypes, as those shown in the Results section.
Data Availability
All data gathered and reported in this study are available as supplementary material. The two datasets of images and annotations described in the “Datasets” section, ar available at https://tinyurl.com/ChronoRootSupplementary. Supplementary figures and tables referenced in this work, as well as a detailed description of the hardware system are available in the Annex. The 3D printing and laser cutting files are included in the Supplementary File.
Code Availability
The source code corresponding to ChronoRoot, namely the deep learning model and the graph generation procedures, is publicly available at https://github.com/ngaggion/ChronoRoot
The source code corresponding to ChronoRoot imaging controller, namely the web interface to check and set up the image acquisition parameters, is publicly available at https://github.com/ThomasBlein/ChronoRootControl
Author contributions
TB, MC, EF and FA conceived the project. NG and EF designed the deep learning models. NG implemented the deep learning models, ran the numerical experiments and generated the figures. TB, VD, EL and SL designed and built the hardware system. TB and VD implemented the web control interface. TR prepared the plates with Arabidopsis seeds and launched the experiments for image acquisition. AC and NG annotated the images used to train the deep learning models. NG, EF, FA, DM and TB analyzed and interpreted the results. EF, FA, TR, DM, MC, TB, NG wrote the paper.
Competing Interests
The authors declare no competing interests.
Annex
CNN Architectures
Hardware Description
In this section we include supplementary notes with additional details about the hardware specification. These notes are provided as open-hardware specifications, to encourage other scientists to 3D print and mount the device in their own laboratories.
Acknowledgements
We would like to thank Fablab Digiscope | LRI | UPSACLAY, and in particular Romain Di Vozzo, for fruitful discussions, his advice in the design and for the access to their digital fabrication equipment. We thank Jean-Paul Bares and Maël Jeuffrard from IPS2 for support and assembling of the ChronoRoot modules. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp used for this research. This work was supported by grants from French State (Saclay Plant Sciences, reference n° ANR-17-EUR-0007, EUR SPS-GSR) managed by the French National Research Agency under an Investments for the Future program (reference n° ANR-11-IDEX-0003-02) to VD, TR, MC and TB; CNRS through the MITI interdisciplinary programs to TB; AXA Research Fund, ANPCyT (PICT2018-3907) and UNL (CAI+D 50220140100084LI.) to EF; ANPCyT (PICT2016-0007) to FA; ANPCyT (PICT 2018-3384) to DM; and CNRS (Laboratoire International Associé NOCOSYM) to MC and FA. FA, DM and EF are researchers of CONICET; NG and AC are fellows of the same institution. TB and MC are researchers and VD is an engineer of CNRS. EL and SL are technicians and TR is a fellow of University Paris-Saclay.
Footnotes
↵iv SegNet publicly available implementation: https://github.com/aizawan/segnet/blob/master/segnet.py
↵v ChronoRoot source code available at: https://github.com/ngaggion/ChronoRoot
↵vi CRF source code available at: https://github.com/lucasb-eyer/pydensecrf