Abstract
In solid tumors, elevated fluid pressure and inadequate blood perfusion resulting from unbalanced angiogenesis are the prominent reasons for the ineffective drug delivery inside tumors. To normalize the heterogeneous and tortuous tumor vessel structure, antiangiogenic treatment is an effective approach. Additionally, the combined therapy of antiangiogenic agents and chemotherapy drugs has shown promising effects on enhanced drug delivery. However, the need to find the appropriate scheduling and dosages of the combination therapy is one of the main problems in anticancer therapy. Our study aims to generate a realistic response to the treatment schedule, making it possible for future works to use these patient-specific responses to decide on the optimal starting time and dosages of cytotoxic drug treatment. Our dataset is based on our previous in-silico model with a framework for the tumor microenvironment, consisting of a tumor layer, vasculature network, interstitial fluid pressure, and drug diffusion maps. In this regard, the chemotherapy response prediction problem is discussed in the study, putting forth a proof-of-concept for deep learning models to capture the tumor growth and drug response behaviors simultaneously. The proposed model utilizes multiple convolutional neural network submodels to predict future tumor microenvironment maps considering the effects of ongoing treatment. Since the model has the task of predicting future tumor microenvironment maps, we use two image quality evaluation metrics, which are structural similarity and peak signal-to-noise ratio, to evaluate model performance. We track tumor cell density values of ground truth and predicted tumor microenvironments. The model predicts tumor microenvironment maps seven days ahead with the average structural similarity score of 0.973 and the average peak signal ratio of 35.41 in the test set. It also predicts tumor cell density at the end day of 7 with the mean absolute percentage error of 2.292 ± 1.820.
Author summary The disorganized structure and leakiness of tumor vessels induce the inadequate blood supply and high fluid pressure within tumors. These features of the tumor microenvironment, identified as delivery barriers, lead to an insufficient amount of drugs to reach the interior parts of tumors. It is observed that the use of anti-vascular drugs makes the structure and function of the tumor vascular system more normal. Moreover, the combination of these drugs with cytotoxic agents provides favorable results with increased treatment response. But, it is also important to adjust the treatment schedule properly. In this regard, we build a deep learning model, designed to examine the tumor response with the ongoing treatment schedule. Our study suggests that deep learning models can be used to predict tumor growth and drug response in the scheduling of cytotoxic drugs.
Introduction
Tumors need their blood supplies to grow beyond the size of 1-2 mm3 in diameter and meet the needs of oxygen and other nutrients. For this reason, tumors stimulate angiogenesis, a process in which tumors form the new blood vessels from pre-existing ones by secreting the various growth factors and, most importantly vascular endothelial growth factor (VEGF). The angiogenic switch between proangiogenic and antiangiogenic factors is activated for tumor progression and metastases [1]. Due to tumor-induced angiogenesis, the newly developed vessels have a leaky and disorganized structure accompanying a microenvironment, identified by hypoxia, acidosis, and increased fluid pressure [2]. As a result, this structurally and functionally abnormal tumor vascular network leads to the heterogeneous and inadequate drug distribution inside tumors.
The chaotic architecture, high vascular permeability of tumor vessels, and the lack of functional lymphatics lead to elevated interstitial fluid pressure (IFP), creating a barrier for the transport of therapeutic agents and nanoparticles [3,4]. IFP is uniform across the tumor nearly equal to microvascular pressure (MVP), but it drops precipitously at the tumor boundary. Therefore, the absence of a pressure gradient along vessels hampers the penetration of cytotoxic drugs to the interior parts of tumors transported by convection [3].
Normalization of tumor vasculature by using antiangiogenic agents is a widely used treatment modality in cancer therapy. Antiangiogenic agents maintain the balance between proangiogenic and antiangiogenic factors similar to healthy tissues. Furthermore, these agents normalize the structure and function of tumor vascular network transiently by inducing reduced vessel diameter and decreased vessel wall permeability. This process triggers tumor vessels to be more useful for the delivery of drugs as well as oxygen and nutrients to the targeted cancer cells [2,5]. Vascular normalization improves the convective transport of drug particles with a decrease in IFP, thereby inducing pressure gradients across vessel walls [1,6].
Antiangiogenic agents can act directly on the tumor vasculature [7]; also, several preclinical and clinical studies show that the application of antiangiogenic agents together with chemotherapy drugs provide beneficial results with increased therapeutic outcome [8–10]. The combination therapy can enhance the delivery of therapeutic agents to the interior parts of tumors [6]. Normalization is a transient process that means there is a time window for vessel normalization to occur. For this reason, chemotherapy drugs should be carefully administered within this window to benefit from the improved vessel conditions [11]. Moreover, the excessive application of antiangiogenic agents for more extended periods brings about vessel pruning, which decreases the outcome of combined therapies [12, 13]. Therefore, the appropriate timing and dosing of the agents should be carefully adjusted to improve the functionality of blood vessels as well as the delivery of anticancer drugs to tumor cells.
Mathematical models are commonly studied in cancer therapies to simulate the delivery of cytotoxic drugs and understand the relations between tumor microenvironment and drug delivery. Different approaches to the modeling of tumor vasculature and angiogenesis have been suggested to study antiangiogenic agents’ applications combined with chemotherapy drugs. By using discrete vasculature models, the delivery of chemotherapy drugs to tumors and the treatment response have been investigated extensively [14–17]. Also, in 2D and 3D blood flow models, the use of antiangiogenic agents and their effects on tumor response have been studied by Stephanou et al. [18]. Normalization is also simulated to investigate its effects on blood flow and the combination of antiangiogenic agents with cytotoxic drugs [13,19–21]. In addition to mathematical models, angiogenesis imaging is largely studied to examine tumor growth, characterization of tumor vasculature, and the response of the therapies. Employing various imaging modalities such as computer tomography (CT) and magnetic resonance imaging (MRI) gives an insight into the progress of tumor and vascular structure [22,23]. Additionally, photoacoustic imaging is exploited to monitor tumor angiogenesis due to its higher contrast and spatial resolution [24–26]. Besides, it is feasible to detect the early tumor growth and vascularization and observe the progression of antiangiogenic treatments [27, 28].
Early tumor response prediction, a measure of the effectiveness of treatment, is crucial in anticancer therapies to determine appropriate treatment schedules, apply the optimal drug dosages to patients, and increase patient survival. In this regard, to develop successful models for tumor response prediction, it is vital to monitor the distribution of cytotoxic drugs and to evaluate the tumor response to therapies in advance. Response prediction problems are often formulated by developing mathematical models benefiting from reaction-diffusion frameworks [29–31]. These models are capable of modeling tumor response to an extent, but they are generally deterministic models driven by a limited number of parameters that might constrain the model to comprise the inherent tumor growth patterns. However, several studies in deep learning outperform the traditional approaches in various tasks such as tissue classification, tumor segmentation, and tumor growth prediction [32–35]. A convolutional neural network (CNN) is developed by Urban et al. to classify the images of vascular networks taken before and after various antiangiogenic drug applications in vitro [36]. In another study led by Ha et al., a CNN algorithm is used to determine the chemotherapy response prediction in patients with breast cancer. The breast MRI dataset is utilized as a baseline, and the treatment response before the application of chemotherapy drugs is predicted [37]. Positron emission tomography (PET) and CT images from different types of cancer are benefited in recent studies to develop CNN models that have the potential to predict the response of chemotherapy with a high sensitivity [38, 39].
Deep learning allows us to build computational models to learn representations of data with multiple levels of abstraction for the given task [40]. For instance, a feed-forward neural network is a function approximation that maps input data to output data. The function is formed by composing simpler non-linear functions where each function provides a new representation of input data [41]. After the model extracts the information out of the input data, it amplifies the essential features for a given task such as classification, segmentation, and regression. Convolutional neural networks, a neural network type, perform convolutions over the input by using the given number of filters, which are capable of learning the spatial and temporal dynamics in image data [40]. They are robust to overfitting due to the convolution operation properties, which reduces the full connection of the network. These models have applications in image [42] and video recognition [43], medical image analysis, and image processing [44]. By employing CNNs, models can be created for spatial and temporal forecasting problems in an end-to-end manner in the presence of sufficient data.
Tumor response prediction can be formulated as a spatio-temporal forecasting problem under the effects of interventions. Interventions can be described as drug injections. Therefore, the forecasting problem is equivalent to modeling the tumor microenvironment during the ongoing treatment. The tumor microenvironment can be described with tumor density, vasculature, interstitial fluid pressure (IFP), antiangiogenic treatment, chemotherapy maps, and drug dosages from the ongoing treatment. The model, denoted by F, takes tumor microenvironment maps Xt and drug scalars St as inputs at time t and predicts future tumor microenvironment maps Xt+1 at time t + 1. The predicted future tumor microenvironment maps allow us to investigate tumor growth and shrinkage patterns as well as drug diffusion maps, which are the key indicators of treatment efficiency.
In this study, we develop a deep learning model, which can capture the tumor response behavior conditioned by the ongoing treatment schedule. Our study suggests that deep learning models can be used to predict tumor growth and drug response in the scheduling of cytotoxic drugs. Since the required input data such as tumor density, IFP, vasculature, and drug maps is hard to collect from clinical patients for various reasons, we use the synthetic data from the mathematical model built in our previous paper [21]. Therefore, the proposed deep learning model F encapsulates the non-linear partial differential equations (PDEs) that govern the spatio-temporal dynamics of the tumor microenvironment. The model consists of multiple CNNs. Our motivation for using CNNs is based upon two key reasons; CNNs can extract coupled spatial features from multichannel inputs, and they can utilize the spatial features to predict future microenvironment maps. In the end, our model uses tumor microenvironment channels and drug scalars as inputs and predicts future tumor microenvironment maps which may assist to find appropriate scheduling and dosages of the combination therapy for the patients.
Results
Our deep learning model aims to predict chemotherapy response by outputting future tumor microenvironment maps. The proposed model, Diffuser-Elapser Network (DENT), consists of seven submodels where each of them is a CNN. All submodels are trained separately as their tasks are different from each other. In the training, we perform cross validation among the unique patients. Therefore, we ensure that there is not any dependency between training and validation sets. Each submodel contains several convolutional layers. Model specifications and training procedures can be found in Methods section.
The DENT model takes input tensors describing the current tumor microenvironment state, which consists of five channels, namely tumor density, vasculature, IFP, antiangiogenic drug, and chemotherapy drug maps that are accompanied by chemotherapy and antiangiogenic drug dosages and predicts the future tumor microenvironment maps. The model can use its predictions as its next inputs. Since the model predicts future tumor microenvironment maps, we use two image quality evaluation metrics, which are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) [47]. PSNR represents a measure of the quality of prediction by calculating the ratio between the square of maximum fluctuation among pixels in the ground truth image and MSE between ground truth and predicted images. SSIM measures the perceptual difference between two images in terms of structural information in the images. In this method, ground truth images are considered as references that have the reference quality, and the quality of another image is measured by comparing it with the initial one. SSIM is mostly applied to improve or track the perceptual metrics based on the structural information; on the other hand, PSNR is based on estimating pixel-wise error. Both metrics are unitless quantities. The PSNR score approaches infinity as the MSE approaches zero. The range for an SSIM score is the interval of [−1,1]. A higher score denotes a better prediction quality in both metrics.
In the study, we first investigate the performance of the model on predicting future tumor microenvironment maps. To give an insight, we pick a synthetically generated case that contains a patient’s tumor microenvironment maps during treatment. The tumor microenvironment maps consist of five channels correspond to tumor cell density, vasculature, IFP, antiangiogenic drug diffusion, and chemotherapy drug diffusion maps. The input, ground truth, and predicted tumor microenvironment maps during the treatment are presented in Fig 1. We prefer eight days-long cases so that we can compare our predictions with the previous simulation findings [21], which is limited to simulating 8 days after the initiation of therapy. The patient receives treatment between day 21 and day 28. The application of the antiangiogenic drug is at the end of days 21, 23, 25, and 27, and the chemotherapy drug is at the end of days 23, 25, and 27. Our model takes the tumor microenvironment maps just before the initiation of therapy as input and predicts the future tumor microenvironment maps by taking its outputs as new inputs. The predicted tumor microenvironment maps show the effects of ongoing treatment on the tumor microenvironment. A qualitative assessment of Fig 1 shows that our model successfully predicts future tumor density, vasculature, IFP, antiangiogenic drug, and chemotherapy drug maps during ongoing treatment.
In the experiments, we use 135 test cases from three different patients. We have included five different therapy initiation days and nine different dosage combinations of antiangiogenic and chemotherapy drugs in the cases. Therefore, the cases differ by the treatment initiation days, applied drug dosages, and patient characteristics. A case corresponds to a tensor of 8 days-long tumor microenvironment maps sized 151 × 151 × 8 × 5 (Width x Height x Time Dimension x Channels) accompanied by drug scalars 8 × 2 generated synthetically by using a mathematical model [21]. The consecutive frames correspond to time steps from 0 to 7 and are separated by a day. The five channels represent tumor cell density, vasculature, IFP, antiangiogenic drug diffusion, and chemotherapy drug diffusion maps. Drug scalars denote the dosage for drug insertion in each time step if there is. The insertion schedule is the same for all cases; antiangiogenic drug is inserted at the end of time steps 0, 2, 4, and 6 and chemotherapy drug is inserted at the end of time steps 2, 4 and 6.
By using completely new cases to test our model, we assure that the model does not see the test data before. Each forward step of the DENT model is trained to predict tumor maps at the next time step which is a day ahead. The average PSNR and SSIM scores are shown in Fig 2 over the test set. We see that our model handles to predict future tumor microenvironment maps up to seven days ahead with the average structural similarity score of 0.973 and the average peak signal ratio score of 35.41. The decreasing trends in SSIM and PSNR scores over time are expected as the prediction error is accumulated in each forward step in time. We also observe drops in average SSIM and PSNR scores at time steps 3, 5, and 7 which correspond to one day ahead predictions following the drug injections. Therefore, a possible reason behind these fluctuations may be the challenges in the drug insertion processes.
We next examine if the model encapsulates the tumor growth and shrink patterns during the treatment by calculating a dimensionless metric called tumor cell density, C. Tumor cell density over the whole tumor shows the effectiveness of ongoing treatment. It is calculated by averaging the pixel values after thresholding the pixels of the tumor density map as performed in the previous study [21]. We compare the tumor cell density of ground truths, Cgt, and predictions, Cpred, over the time steps. For the selected patient case in Fig 1, ground truth and predicted tumor cell density values during treatment are presented in Fig 3. We calculate the absolute percentage error between Cgt and Cpred for 135 test cases which are shown in Table 1. The mean absolute percentage error at the end of day 7 is 2.292 ± 1.820 which shows that our model encapsulates the tumor growth and shrink patterns under the effects of ongoing treatment. It can be seen that the standard deviation of error increases over time since the error accumulates in each prediction.
Discussion
In this study, we build a deep learning model designed to solve the chemotherapy response prediction problem. The model is trained with simulation data, which includes tumor growth and angiogenesis, reflecting the biological aspects of tumor microenvironment [20,21]. Here, the proposed deep learning model can predict chemotherapy’s response to tumor structure over an indefinite period via iterating for an indefinite number of times, permissively. Our model operates on a total of five channels: the maps of tumor cell density, vasculature, interstitial fluid pressure, and antiangiogenic and chemotherapy drug diffusions. The predicted maps show the spatial and temporal effects of ongoing treatment on tumor microenvironment maps. The model can take its outputs as inputs, go further in time, and predict the following state of the tumor by iterating this process as many times as required. In these iterations, there is an option of drug injection fed into the model as an input, if it is in the treatment schedule. To include the injection step in the model, first, the drug diffusion maps on the tumor structure are predicted, then these maps are used for the next tumor state prediction in combination with the other inputs such as tumor cell density, vasculature, and interstitial fluid pressure maps. To evaluate the predictive capabilities of our model, we track PSNR and SSIM scores calculated between the ground truth and predicted tumor microenvironment maps during the treatment. Our model can predict future tumor microenvironment maps up to 7 days with the average SSIM score of 0.973 and the average PSNR score of 35.41 at the end of day 7. We have assessed the tumor cell density among the entire predicted tumor density Cpred and compared it with the ground truth cell density Cgt. To obtain the values of cell density both for the predicted tumor and the ground truth tumor, we have used the same method as [21], merely averaging the pixel values that are above a threshold value. We use the percentage error between Cpred and Cgt to evaluate the performance. The model predicts tumor cell density at the end of day 7, with the mean absolute percentage error of 2.292 ±1.820.
Chemotherapy response prediction aims to successfully predict whether the patient will respond to the treatment, depending on the tumor characteristics of the patient. Current approaches to chemotherapy response prediction are mainly based upon classification models. A recent study [48] focuses on the prediction of lung cancer treatment response for patients treated with chemoradiation. By analyzing time-series CT images of patients, the survival probability and cancer-specific outcomes have been predicted. In another study by Ha et al. [37], it is aimed to predict the effectiveness of neoadjuvant chemotherapy (NAC) response by using a breast MRI tumor dataset. Patients are classified into three groups as complete, partial, and no response, based on their response to NAC treatment. Although these classification-based models predict chemotherapy response to some extent, their predictions lack spatial and temporal information, which provides the necessary explanations behind predictions. For instance, providing maps of future tumor microenvironment states after the treatment may give clinicians an idea for re-planning the treatment scheduling process. Therefore, there is a challenge in building models that provide detailed information in the chemotherapy response prediction task, which brings us to the modeling of the tumor microenvironment.
On the other hand, mathematical models rely on nonlinear PDEs that simulate the dynamics of the tumor microenvironment, considering the effects of treatments. These models enable us to observe the diffusion of the chemotherapy drug to tumor cells via the normalized vasculature network, and the effectiveness of the applied treatment by producing maps of the future tumor microenvironment states. This approach may eliminate the main drawback of the class-based chemotherapy response prediction models, which is a lack of insight behind the predictions. A recent study [21] presents a mathematical model to simulate the tumor state according to the previous tumor state and the drug diffusion inside the tumor. In each iteration of the model, a predetermined theoretically-thirty-minute gap period proceeds, and the subsequent tumor state is calculated. This allows the model to output the tumor states with a time-resolution of thirty minutes, which is suitable to study the tumor shrinkage or growth comprehensively under the effects of ongoing treatment. The model is deterministic, which means differential equations, predetermined biophysical factors, and constraints are utilized. In potential clinical use, this could hinder the model from making correct interpolations, since these factors and constraints could be varying from patient to patient contrary to the fixed values used in the simulation. Capturing these factors and constraints from patient data is essential to predict patient-specific chemotherapy response. Calculations in the simulation are at the pixel level, and the model cannot capture the detailed physical structure of the tumor and recognize any particular situation, such as detecting the tumor type or distinguishing the smaller sections of the tumor. Since the coefficients used in the differential equations are constant and non-learnable, a deterministic model could not learn from clinical data and be precise enough for clinical use, making it impossible to generalize over thousands of patients.
In the wake of these limitations in the literature, we propose a deep convolutional model. The model simulates tumor microenvironment maps, which encapsulate spatio-temporal effects of the ongoing treatment such as tumor growth and shrinkage patterns, together with the drug diffusion maps. The model outputs future tumor microenvironment maps, which fill the explanatory information gap in classification-based response prediction models. The sequence of predicted maps indicates the response to the therapy in a daily regime. The model predictions reveal the effectiveness of therapy schedule and drug dosages on patients before the application of the therapy. This could allow better adjustment of schedules, use of the potent schedules, and elimination of the ineffective ones. Although mathematical models can simulate tumor microenvironment, they tend to neglect patient-specific tumor response patterns due to a limited number of parameters, fixed coefficients, and deterministic rules governing the model. Since the proposed model is built with end-to-end learning, it can learn patient-specific tumor response patterns in the presence of sufficient data. The model may improve the personalization of treatment by extracting specific clinical features of a patient and making predictions based on them.
Our model for the chemotherapy response prediction task has potential clinical uses; however, some limitations can be eliminated with further improvements. A limitation of the proposed model is using 2D tumor microenvironment maps instead of 3D maps, which can make better use of the tumor microenvironment’s spatial information. When the tumor microenvironment maps are missing the 3rd spatial dimension, the true nature of the tumor cannot be reflected in detail, leading to an incomplete representation of the tumor microenvironment within the model. Therefore, 3D mathematical models of tumor [49] or 3D scanning methods such as CT are the primary candidates of data sources to solve the incomplete environment representation problem. In addition to the 3rd spatial dimension, tumors also have a continuous structure across the time dimension, that is, a continuous course of growth and shrinkage. Modeling the time dependencies between the sequential states of the tumor may be necessary, assuming that the future tumor growth or shrinkage pattern follows its past trend under constant conditions. The input should consist of temporal information of the tumor microenvironment maps to encapsulate the effects of time dependencies on a tumor microenvironment. The model can utilize the input sequence to reveal the features of patient-specific tumor dynamics. These features can be critical to include patient-specific tumor growth patterns and tumor aggressiveness in model predictions. Although our CNN-based model captures spatio-temporal dependencies to some extent, it is preferred to use long short-term memory (LSTM) [50] networks with sequential inputs, combined with CNNs. Convolutional LSTMs [51] can be utilized to extract both spatial and temporal dynamic changes in a single network as they are already exploited in the tumor growth prediction problem [52]. The main challenge of this approach would be to acquire regularly timed and spatially aligned scans to generate sequential data.
The simulated cases used in our study differ by therapy initiation day, drug combinations used in the therapy, and unique patients imposed by the randomly initialized vasculature maps. However, all simulated cases share the same insertion schedule in which antiangiogenic drug therapy is initiated at the beginning of the therapy, and chemotherapy is started with the delay of two days. Both drugs are inserted every other day. Since our cases lack the variations in the insertion schedules, our model may be limited in encapsulating the effects of longer or shorter time intervals between the initiation of antiangiogenic and chemotherapy treatment or non-overlapping insertion days of two types of drugs. A possible solution is incorporating various realistic insertion scheduling scenarios in the cases.
Obtaining the training data from simulations is beneficial in a proof of concept study; however, for a distinguished and genuinely working model, using clinical training data is inevitable. In order to make sure that a deep learning model can simulate and generalize chemotherapy response in a real scenario, it is necessary to train a model with a reasonably large amount of clinical data with sufficient variations of treatment schedules. A clinically applicable response prediction model’s crucial feature is its ability to make predictions in the same domain with clinical data such as MRI, CT, and PET scans. Each scanning technique emphasizes a particular aspect of a biological structure, capturing a unique type of information about the tumor microenvironment. By using these scanning techniques, it is possible to obtain the maps of tumor cells, tumor vasculature [53], IFP [54] and cytotoxic drug distributions [55, 56]. Various scanning techniques should be combined to feed the model’s input to represent the tumor to the full extent, thereby introducing as many aspects of the tumor as possible. This diversity of scan types would provide a complete representation of the tumor within the model, boosting prediction and simulation capabilities.
Methods
The mathematical model for acquiring synthetic data
In this study, a mathematical model of tumor built in our previous study [21] is used to obtain the training and testing data sets. Non-linear PDEs are written in dimensionless form and utilized in the model to mimic the tumor and its microenvironment biologically.
The model incorporates tumor cell density and vasculature as well as their interplay and consists of tumor cell density, vasculature, IFP, antiangiogenic agent, and chemotherapy drug. The reaction-diffusion equations are used to describe the spatio-temporal distribution of tumor cell density and vasculature (Eqs 1 and 2). Tumor cell density and vasculature are denoted by n(x,t) and m(x,t), respectively. In Eq 1, the first term on the right-hand side defines the diffusion of tumor cell density, where Dn is the diffusion coefficient, the second term models the tumor growth rate, where r is the growth rate and nıim is the maximum carrying capacity. The third term couples the tumor cell density and vasculature in which αmn indicates the proliferation rate of tumor cells when tumor vessels are present, and the fourth term describes the relationship between the tumor cell density and the chemotherapy drug, here dr is the elimination rate of tumor cells when chemotherapy drug is applied. Initially, tumor cells are assumed to be distributed by Gaussian.
Distinct to healthy vessels, tumor vasculature is a heterogeneous structure, having vessels with tortuous and larger pores. In the model, the Eq (2) is used to represent the heterogeneous tumor vascular network, and it involves the terms for the diffusion of vessels, the production of islands of vessels, the directed motion of vasculature to tumor cells, the production of tumor vessels due to tumor-induced angiogenesis and the elimination of vessels by application of antiangiogenic agent, respectively. For the initial state of tumor vessels, it is assumed that tumor vessels have a random and positively distributed configuration.
To describe IFP in the solid tumor, the Eq (3) is used: where the term λbm(x,t) [Pv— P(x, t) — σv(πc—πi)] is the fluid source from blood vessels to the interstitial space and λℓP (x, t) is the drainage of fluid from interstitial space to lymph vessels. In the equation, λb and λℓ are the hydraulic conductivities of blood and lymp vessels, respectively. The parameter Pv is the vascular pressure, P is the interstitial fluid pressure, σv is the osmotic reflection coefficient. The terms πc and πi indicate the capillary and the interstitial oncotic pressures.
The transport of the antiangiogenic agent is represented by the Eq (4). Here, the first term on the right-hand side is the diffusion of an antiangiogenic agent, where DA is the diffusion coefficient of the antiangiogenic agent in tissue. The second term is the diffusion of the agent through vessels where λA is the transvascular diffusion coefficient of the antiangiogenic agent, and Av is the concentration of the agent in plasma. The third term is the drainage of agents into the lymphatics, and the last term is the natural decay of agents in tissue, where kA is the decay rate of antiangiogenic agents.
A convection-diffusion equation is used to present the delivery of chemotherapy drug molecules, which are larger particles (around 100 nm in size). In the Eq (5), the first and second terms define the diffusion and convection of chemotherapy drugs in the tissue where Dd is the diffusion coefficient of drugs, and kE is the retardation coefficient for convection in the interstitium. The third term is the convection of drugs through the vessels, where σd is the solvent drag reflection coefficient, the fourth and fifth terms are the drainage of chemotherapy drugs to the lymph vessels and the reaction of chemotherapy drugs with tumor cells, where dr is the reaction rate. The last term represents the decay of drugs in tissue, where kd is the natural decay rate.
The Eqs (4) and (5) are solved in steady-state since the time scale for the transport of antiangiogenic agent and chemotherapy drug is shorter than the time scale of the tumor growth. Antiangiogenic agent and chemotherapy drug are applied with bolus injection through an exponential decay function: where A0, d0, and stand for the initial plasma concentration and the plasma half-life of antiangiogenic agent and chemotherapy drug, respectively. No-flux boundary conditions are applied for antiangiogenic agent and chemotherapy drug.
Datasets
The datasets used in this study generated synthetically by a MATLAB simulation [21] which models tumor microenvironment mathematically as it is described in the previous section. The simulation results are consistent with experimental studies, allowing them to be used for data generation [57–61]. Two datasets are used in the training of the proposed deep learning model DENT. In the training and validation sets, cases from five unique patients are used. A case corresponds to tumor microenvironment maps during eight-day combination therapy treatment. The cases contain tumor microenvironment maps with a time-resolution of 30 minutes. The cases have five different therapy initiation days in the range of day 14 to day 21. The combinations of three different antiangiogenic and chemotherapy drug dosages are used in treatments. The insertion schedule is the same in all treatments. The antiangiogenic drug is inserted at the end of days t0, t2, t4, t6 and chemotherapy drug is inserted at the end of time steps t2, t4, t6 where t0 denotes the starting day of therapy. Each patient has 45 cases considering the initiation days and drug dosage combinations. Training and validation datasets are formed by utilizing 225 synthetically generated cases with five unique patients.
The first dataset, employed by Diffuser, contains drug map changes in the presence of drug insertion, a combination of antiangiogenic and chemotherapy drugs (see Fig 4).
This dataset consists of 900 pairs. The first element is the tumor microenvironment tensors accompanied by drug scalars that represent the dosage of the given drug. The second element is the drug diffusion maps tensor immediately after drugs are diffused. The size of tumor microenvironment maps in pairs is 151 × 151. It should be noted that we only consider changes in drug diffusion maps between input and target tensors. Therefore, the time interval between the input and output tensors corresponds to an hour in which drugs are fully diffused in the tumor microenvironment. Using these pairs in the Diffuser dataset, we train our model to approximate another mapping function between input and target tensors, which is equivalent to learning dynamics of drug diffusion given the tumor microenvironment maps and drug dosage scalars. Since there is not a considerable change in tumor density, vasculature, and IFP maps in an hour, we omit these channels in the target tensor.
The second dataset, used by Elapser, contains the changes in the tumor microenvironment in the absence of drug insertion between input and target tensors (see Fig 5). The dataset consists of 1575 well-separated pairs of tumor microenvironment maps. The first element of the pair is the input tensor, whereas the second element is the target tensor. Both of the tensors comprise five channels sized by 151 × 151, which are tumor density, vasculature, IFP, antiangiogenic drug, and chemotherapy drug maps. Each pair of tensors is separated by a day apart. By using these pairs, we train our model to approximate a mapping function that maps input tensors to target tensors. Since the time interval between the input and target tensors corresponds to a day, the task is equivalent to learning the dynamics of tumor states in a daily regime.
Data Preprocessing
We aim to eliminate any numerical instabilities among the pixel values of input channels and drug scalars. All input images and scalars in the dataset are scaled with min-max scaling by using the following function:
In the case of image inputs, such as tumor density, vasculature, IFP, chemotherapy, antiangiogenic maps, xi is the pixel value, max(x) is the maximum pixel value and min(x) is the minimum pixel value in the interesting channel. In the case of scalar inputs, such as antiangiogenic and chemotherapy drug dosages, xi is drug dosage value, max(x) is the maximum value, and min(x) is the minimum value from corresponding drug. The normalized value of xi corresponds to zi. After the scaling, each pixel and scalar input is within the range of [0,1]. It should be noted that each max(x) and min(x) values are obtained from the training set to ensure that we do not use any information obtained from the test set.
The Deep Learning Model
Functionality of Simulating an Indefinite Period of Time
The proposed model takes the tumor microenvironment maps Xt and drug scalars St and predicts the future microenvironment maps Xt+1 in each forward pass. The time difference between t and t + 1 corresponds to a day. Since Xt and Xt+1 are both tumor microenvironment maps, we can feed the model with its prediction Xt+1 and St+1 to generate Xt+2. This process can go on indefinitely to obtain the tumor state at any time point Xt+x, where x is arbitrary and shows the number of prediction steps.
DENT model as the composition of Elapser and Diffuser models
The DENT model has the task of predicting future tumor microenvironment maps considering the effects of ongoing treatment. The model consists of multiple CNN submodels in order to extract spatial features from multichannel tumor microenvironment inputs. The model utilizes these features to predict future tumor microenvironment maps, which is equivalent to learning spatio-temporal dynamics of the tumor microenvironment. The model has two main tasks; the first task is inserting and diffusing the given drug dosages in the given tumor microenvironment, and the second task is predicting future tumor microenvironment maps. Since these tasks are not overlapping, we build two submodels called Diffuser and Elapser networks, which work independently. These two models are integrated to attain the DENT network, which can iterate with taking its outputs as inputs. It is able to check if there is any injection and interfere with the iteration process in case of necessity.
The first part of the proposed model, the Diffuser network, contains two independent CNN submodels, which inserts and diffuses the scalar drug dosages given together with the input tumor microenvironment maps, generating a new drug diffusion maps. It is a conditional network which is initiated if scalars are different from zero. In the case of zero drug dosage, the Diffuser network outputs the relevant drug map without any change. The second part, the Elapser model, contains five independent CNN submodels, which takes tumor microenvironment maps as input tensor and predicts future tumor microenvironment maps in a channel-wise manner. The Elapser model encapsulates the changes in each channel so that it can extrapolate to the future.
By creating a pipeline with the Diffuser and Elapser networks as shown in Fig 6, we build a deep learning model that iterates for a given number of time steps. It takes its outputs as the inputs of the next step while iterating, which is suitable for performing intervention due to drug insertion schedule.
Model Specifications and Training Procedures
As previously mentioned, the proposed pipeline consists of two submodels. The first submodel, Diffuser Network, injects and diffuses the scalar drug dosage given the input tensor, generating new antiangiogenic and chemotherapy drug diffusions maps. An input tensor for the Diffuser Network contains tumor microenvironment maps accompanied by two dosage scalars, as shown in Fig 4. We first tile these scalars to our input tensor so that the input has the shape of 7 × 151 × 151. The Diffuser network includes two CNNs that are trained separately with the same input tensors. The first CNN approximates a new antiangiogenic drug diffusion map, and the second CNN approximates a new chemotherapy drug diffusion map. The second submodel, Elapser Network, forecasts the future tumor microenvironment maps by extrapolating from the input tensor shaped 5 × 151 × 151 shown in Fig 5. It outputs the future tumor microenvironment maps as a five-channel image tensor. There is a separate CNN block for predicting each channel of future image tensor.
All CNN submodels are trained separately as their tasks are different from each other. In the training, we use 225 synthetically generated cases from five unique patients. We perform 5-fold cross-validation among the five patients. Therefore, there is not any spatial or temporal dependency between training and validation sets. The submodels are trained with MSE loss for 200 epochs with early-stopping to avoid over-fitting. The activation function is ReLu and kernel size fixed as 5 × 5, whereas stride length is 1. After each layer, 2D batch normalization is performed except the final layer. We use ADAM [45] optimizer with a batch size of 10 and initial learning rate of 10−4. The number of layers and filters are selected based on the best performance, and increasing the depth of network or number of filters does not affect the model performance. The model specifications are shown in Table 2. All models are built with Pytorch [46] backend and trained on a single NVIDIA Tesla K80 GPU.