Deep-learning based three-dimensional label-free tracking and analysis of immunological synapses of chimeric antigen receptor T cells

We propose and experimentally validate a label-free, volumetric, and automated assessment method of immunological synapse dynamics using a combinational approach of optical diffraction tomography and deep learning-based segmentation. The proposed approach enables automatic and quantitative spatiotemporal analyses of immunological synapse kinetics regarding morphological and biochemical parameters related to the total protein densities of immune cells, thus providing a new perspective for studies in immunology.

Understanding the immune response at cellular scales requires obtaining knowledge of interactions between immune cells and their microenvironment.One of the critical features for studying immunological activity is the immunological synapse (IS), a dynamically interacting junction between immune cells and their targets.Many hierarchical details related to the structural and signalling pathways activated during IS formation have been elucidated by fluorescence microscopic techniques, which have evolved from total internal reflection fluorescence microscopy for interfacial imaging 1 to light-sheet microscopy for volumetric imaging 2,3 .Although such fluorescence-based techniques take a clear advantage of chemical specificity, they are also associated with innate limitations of photo-bleaching, photo-toxicity, and slow imaging, thereby necessitating the use of complementary label-free microscopy methods for evaluating singlecell dynamics 4 .Because the formations dynamics of IS occurs within a few minutes, rapid 4D imaging of immunes cells are required.
Here, we present a method for the systematic analysis of IS of immune cells.To track IS of live immune cells in a labelfree and volumetric manner, three-dimensional (3D) refractive index (RI) tomograms of immune cells are measured using optical diffraction tomography (ODT), a 3D quantitative phase imaging technique 5 .In order to perform automated assessments of IS between immune and target cells, deep learning-based segmentation is employed to the 3D RI tomograms.To validate the method, we studied the IS formation dynamics of chimeric antigen receptor (CAR) T cells.4D RI tomograms of CAR T cells and targets cells are measured at high speed (each tomogram measurement for every 35 seconds) for a long period of time (from 300 seconds to 10 minutes depending on cell types).The formation dynamics of IS of CAR T cells with various intrinsic functionalities are investigated.The dynamic formations of IS are systematically analysed by a deep-learning based automatic segmentation algorithm.

Optical Diffraction Tomography
In order to measure 3D RI tomograms of immune cells, an ODT setup (HT-2H, Tomocube Inc., Republic of Korea) was used.Based on the principle of inverse light scattering, ODT reconstructs 3D RI tomograms of transparent objects, from multiple 2D optical field images 6 .Due to its label-free and quantitative imaging capability, ODT has been used for the study of cell biology 7 , hematology 8 , hepatology 9 , infectious diseases 10 , and cytotoxicity 11 .
The used setup is based on off-axis holograph equipped with a high-speed illumination scanner using a digital micromirror device (DMD) 12,13 .A 2 × 2 single-mode fibre coupler was used to split a coherent, monochromatic laser (λ = 532 nm) into a sample and a reference arm, respectively.The DMD was then placed onto the sample plane of the sample arm to control the illumination angle of the first-order diffracted beam impinging onto a sample.To scan the illuminations at high angles, a 4-f array consisting of a tube lens (f = 250 mm) and a condenser objective (UPLASAPO 60XW, Olympus Inc., Japan) magnified the illumination angle.The light scattered by the samples was then transmitted through the other 4-f array formed by an objective lens (UPLASAPO 60XW, Olympus Inc., Japan) and a tube lens (f = 175 mm).The sample beam was combined with the reference beam by a beam splitter and filtered by a linear polariser.The resultant off-axis hologram was then recorded by an image sensor (FL3-U3-13Y3M-C, FLIR Systems, Inc., USA) that is synchronised with the DMD to record 49 holograms of the sample illuminated with different angles.
Using a phase-retrieval algorithm, the amplitude and phase images can be retrieved from the measured holograms.Based on the Fourier diffraction theorem with Rytov approximation 14,15 , the 3D RI tomogram of the sample was reconstructed from the retrieved amplitude and phase images.To fill up the uncollected side scattering signals due to the limited numerical apertures of objective lenses, the regularization algorithm based on the non-negative constrain was used 16 .The maximum theoretical resolutions of the ODT system were 110 nm laterally and 330 nm axially, according to the Lauer criterion 17 .Finally, the reconstructed RI values were converted into protein densities using the RI increment of  = 0.185 ml/g 18,19 .

Segmentation algorithm
Dataset preparation using the watershed algorithm.To generate the ground truth masks of effector and target cells, we employed a watershed algorithm 20 according to the following steps.First, we processed a raw RI tomogram with four hyper-parameters: (1) initial seed locations for each cell, (2) RI threshold for defining cell boundaries, (3) voxel dilation sizes for merging over-segmented grains into one discrete region, and (4) standard deviation of the Gaussian smoothing mask.The processed tomogram was multiplied to a 3D distance-transform map of the cell regions and segmented by the watershed algorithm.Through iterative adjustment of the parameters, we obtained 236 pairs of 3D tomogram and segmentation masks for effector and target cells.
Automated segmentation strategy using deep learning.The requirement of iteration for parameter tuning of the watershed algorithm-based segmentation method to obtain a single well-segmented label is prohibitive for obtaining a dynamic dataset.Therefore, we employed a deep-learning approach to enable general, high-throughput, and automated segmentation for 3D tomograms.The deep neural network was designed with the main goal of regressing the distance map rather than classifying voxel-wise labels 21 , regarding the following difficulties of our segmentation tasks: (1)  indistinct boundaries between effector-target cell pairs in RI distributions, (2) diverse morphology of cells, and (3) demand for precise segmentation at high resolution.For our training purposes, we converted each label of effector and target cells into distance maps using Euclidean distance transformation, whereas we set the background to zero.Moreover, the effector and target cells were distinguished by their signs on the distance maps (i.e., positive and negative, respectively).The input and output data were 3D RI tomograms and estimated signed distance maps, respectively, with a dimension of 128 × 128 × 64.During the inference, the output distance maps from the network were converted to segmentation masks through simple thresholding.Adopting distance regression improves segmentation accuracy and robustness to overfitting.

Network architectures.
As shown in Supplementary Figure 1, the architecture of our network is based on a 3D UNetlike architecture, with proven good performance for biomedical segmentations 22 .The architecture is composed of a series of contracting and expanding paths; the former includes a series of residual blocks and down-pooling layers, and the latter involves a series of boundary refiners (BRs) and up-pooling layers.The number of filters for the five layers are [32, 64, 128, 128, 256].The feature skip connection of our network passes through the global convolutional network 23 , which was employed to increase the receptive field.Such modifications allowed the network to learn more about the overall cell morphology and critical characteristics of immune cells.
Training protocol.We divided the 236 curated training datasets into 198 training and 38 validation sets, respectively.We chose the L2 loss function setting, which is appropriate for a distance regression scheme.The model was trained with the Adam optimizer (α= 0.5, β = 0.99) using a decaying learning rate (initial value = 0.001).For efficient training, we augmented the data using rotations, resizing, and elastic deformations.The network was trained on four graphics processing units (GPUs; GEFORCE GTX 1080 Ti) for 400 epochs, which took approximately 6 hours.Selection of a model for inference among trained models was based on performance on the validation set.Our network was implemented in Python using the PyTorch package (http://pytorch.org),and the other processing steps were performed in MATLAB.

Cell preparation and establishment of cell lines
Cell lines and culture.The K562 and K562-CD19 cell lines were kindly provided by Travis S. Young (California Institute for Biomedical Research), and cultured in RPMI-1640 medium supplemented with 10% heat-inactivated foetal bovine serum (FBS), 2 mM L-glutamine, and 1% penicillin/streptomycin in a humidified incubator with 5% CO 2 at 37°C.The Lenti-X™ 293T cell line was purchased from Takara Bio, which was maintained in Dulbecco's modified Eagle medium supplemented with 10% heat-inactivated FBS, 2 mM L-glutamine, 0.1 mM non-essential amino acids, 1 mM sodium pyruvate, and 1% penicillin/streptomycin.
To define the IS of CAR-T cells, we generated an mCherry-tagged CD19-BBz CAR construct.The mCherry gene was amplified from pLV-EF1a-MCS-IRES-RFP-Puro (Biosettia, USA), and overlapped with synthetic oligonucleotides of a G4S linker.This PCR product was then overlapped with that of CD19-BBz CAR and inserted into the BamHI and SalI sites of pLV vectors (pLV-BBz-CAR-mCherry).
Two days after stimulation, activated T cells were mixed with the lentivirus supernatant, centrifuged at 1000 ×g for 1 hour and 30 minutes, and incubated overnight at 37°C.CAR-transduced T cells were cultured at 1 × 10 6 cells/mL in RPMI-1640 medium supplemented with 10% heat-inactivated FBS, 2 mM L-glutamine, 0.1 mM non-essential amino acids, 1 mM sodium pyruvate, and 55 μM β-mercaptoethanol in the presence of human recombinant (rh)IL-2 (300 IU/mL) until sorting of CART19 cells from bulk T cells.

Figure 1 Figure 2 .
Figure 1 Flow chart of label-free synapse reconstruction.(a) Data acquisition in optical diffraction tomography (ODT).The experimental setup for ODT is based on a digital micro-mirror device (DMD) for high-speed illumination scanning.Forty-nine holograms at various illumination angles were recorded, and their amplitude and phase delay distributions were retrieved.(b) Synapse reconstruction.A reconstructed refractive index (RI) map (left) was used as an input for our deep-learning model.The model segments CART19 and K562-CD19 cells and defines the immunological synapse.Colour maps are based on the two-dimensional ranges of RI and the RI gradient.(c) Highthroughput segmentation over 0.98 × 1.05 × 0.04 mm 3 .Representative effector-target cell pairs are magnified on the right.

Figure 3 .
Figure 3. Statistical analyses of synapse morphologies depending on the cell-intrinsic functions of CART.(a) Scatterplots of dry masses.(b) (c) Scatterplots of synapse area-per-surface areas Each boxplot indicates the median, upper, and lower quartiles of each population.The attached lines indicate the range of the population.Perpendicular shades indicate the normalized population density distributions.* p < 0.05, ** p < 0.01,*** p < 0.001 by two-tailed Wilcoxon tests.mean ± SD data are available in Supplementary Table 1.