An automatic and efficient pulmonary nodule detection system based on multi-model ensemble

Accurate pulmonary nodule detection plays an important role in early screening of lung cancer. Although there are many presented CAD systems based on deep learning for pulmonary nodule detection, these methods still have some problems in clinical use. The improvement of false negatives rate of tiny nodules, the reduction of false alarms and the optimization of time consumption are some of them that need to be solved as soon as possible. In view of the above problems, in this paper, we first propose a novel full convolution segmentation framework for lung cavity extraction in preprocessing stage to solve the time consumption problem of the existing pulmonary nodule detection systems. Furthermore, a 2D-NestedUNet segmentation network and a 3D-RPN detection network is stacked to get the high recall and low false positive rate on nodule candidate extraction, especially the recall of tiny nodules. Finally, a false positive reduction method based on multi-model ensemble is proposed for the further classification of nodule candidates. Our methods are evaluated on several public datasets, LUNA16, LNDb and ChestCT2019, which demonstrated the superior performance of our CAD system.

Lung cancer is a leading cause of cancer-related death both in men and women. Every 2 year lung cancer results in about 170 million deaths worldwide. The early diagnosis of a 3 lung lesion is recognized as the most important method to reduce the lung cancer 4 mortality rate. The low-dose CT scans which is one diagnostic way can be used to 5 screen for lung cancer in people. Using this screening method can decrease the risk of 6 dying from lung cancer. Now researchers are looking for new attempts to refine CT 7 screening to better predict whether cancer is present [1]. Due to the rich pulmonary 8 vascular structure and the different skill levels of radiologists, the potential malignant 9 lesions are easy to be ignored. In the clinic, an effective way to deal with this problem is 10 to diagnose by two radiologists respectively and then to summarize their answers.
In recent years, with the rapid development of deep learning in the field of medical 17 image analysis, a large number of CNN-based CAD systems have been utilized for the 18 detection of pulmonary nodules [3][4][5]. Compared with the traditional methods [6,7], the 19 DNN-based methods have made great progress in accuracy and practicability of 20 pulmonary nodule detection. Nevertheless, many challenges still exist in the lung nodule 21 detection procedure. The detection procedure is usually divided in three stages, i.e. 22 image prepossessing, nodule candidates extraction and false positives reduction. Firstly, 23 in prepossessing step, how to quickly and efficiently extract lung cavity plays an 24 essential role. Due to the various sizes and morphology of pulmonary cavity, the 25 traditional multi-threshold-values based segmentation is not robust enough. Therefore, 26 offering an efficient lung cavity extraction method is very important. Secondly, in 27 clinical practice, radiologists pay more attention to small nodules, because these small 28 nodules are more likely to cause lung cancer in the future. To ensure high sensitivity for 29 them, in other words, CAD is needed to have a high recall rate for small nodules that 30 may better resolve lung cancer early screening. However, the size of small nodules is too 31 small and their radiographic manifestations always appear ground glass attenuation so 32 that it is likely to be neglected. The presented methods, neither using 2D nor 3D 33 system, can find small pulmonary nodules well [3,4]. That means the developers need 34 invest more research in this area in the future. Thirdly, because abundant tissues exist 35 in the human pulmonary cavity, for instance blood vessel and chest wall, the 36 appearances of these tissues are very similar with the appearance of the pulmonary 37 nodules. This results in producing amounts of false alarms during detection. Making 38 something to accurately distinguish between tissues and nodules thereby reducing the 39 number of false positive signals is very important and necessary [8]. 40 The primary aim of this study is to develop an advanced CAD system that extracts 41 information from medical images efficiently and provides radiologists a precise and 42 timely diagnosis of lung lesion. The key contributions of this paper are summarized 43 below:

44
• we present a FCNN framework which is based on U-Net [12] for quickly and 45 stably performing pulmonary cavity extraction.

46
• we stack a 3D-RPN based detection network [3] and a 2D-NestedUNet based 47 segmentation network [13] for providing target candidates to satisfy high recall 48 rate and low false positive rate of pulmonary nodule detection.

49
• we propose an integrated classification network which consists of ResNet [23], 50 DenseNet [24] and SENet [25], summarizing the results from detection step and 51 classification step to reduce false candidates.

53
Before the rapid development of CNN in medical image processing, the pulmonary 54 nodules detection was mostly based on hardcrafted features. In recent years, with the 55 successful application of deep learning in medical images, the intelligent screening 56 system of pulmonary nodules has been greatly developed. Based on the exploited deep 57 architecture, these approaches can be divided into proprecessing, candidate nodules 58 extraction and false positive reduction. In [3,4,14], the pulmonary cavity of the CT 59 scans was obtained by traditional image processing methods such as region growth and 60 morphological operations. [15] proposed the spot detection for extracting the nodule 61 candidates. Using this method, nodule and non-nodule samples could be distinguished 62 according to their different shape, size, specific texture, density and other features. 63 However, due to the choices of thresholds, the detection performance of this method for 64 April 9, 2020 2/11 small nodules and tiny nodules was very poor and not robust. In recent years, many 65 excellent CNN-based CAD systems for automatic pulmonary nodules detection are 66 presented, which can roughly be divided into 2D image slices and 3D volume images.

67
In [3], 2D image slices were fed into Faster-RCNN [16] to locate suspected nodules and 68 output their sizes in CT images. First from the perspective of 3D volume images, [4] 69 established a pulmonary nodules detection system with the aid of 3D-Faster-RCNN 70 framework. This method could make full use of the spatial information of 3D images.

71
In [17], the attention mechanism is brought into the approach presented in [4] to further 72 acquire 3D spatial semantic information. In [18], a new self-supervised pulmonary 73 nodule detection framework is proposed based on 3D-RPN to improve the sensitivity of 74 nodule detection by adopting multi-scale features to increase the resolution of nodule 75 cubes. Compared with the 2D models, these 3D methods employ spatial semantic 76 information to better analyze the morphological characteristics of nodules and locate 77 them much accurately. However, their recalls for small nodules are not high so that 78 would be easy to increase the false negatives rate for small nodules and tiny 79 nodules. [19] presented a 3D full convolutional network which is based on V-Net [20] to 80 extract nodule candidates. A high recall is the advantage of this solution. For reducing 81 more false positives, [4,6] proposed a simple 3D convolutional network adding the back 82 of detection module to filter false candidates. [21] introduced the attention mechanism 83 after false positive reduction module to facilitae the performance of classification for 84 pulmonary nodules (benign and malignant).

86
Our proposed CAD system can be roughly divided into three stages: 1) FCNN based 87 lung segmentation, 2) multi-model ensemble based pulmonary nodule extraction, and 3) 88 false positive reduction (the whole pipeline shown in Fig 1).
The Overview of Our CAD system. A whole CT scan is fed into our system to predict nodule candidates. The process consists of three stages, i.e. preprocessing, pulmonary nodules extraction and false positive reduction. In first stage, raw data will be processed to extract lung cavity and then fed into second stage to find pulmonary nodule candidates. Finally, the candidates will be filtered in stage 3 to reduce false alarms.
Assume that the input of the segmentation network is a single-channel grayscale 104 image with a size of 1×H×W, denoted as I in , and the output of each convolution 105 module i th in the encoder path is denoted as X i e , which represents the number of output 106 channels, each convolution module i th takes as input to X i−1 e . At the end of the encoder, 107 a feature map F e will be obtained whose size is reduced by 4 times compared to I in .

108
Different from the traditional U-Net structure, we make some modification for our 109 FCNN in the convolution mode of the decoder path, which is to use the same 110 convolution structure as the encoder path to implement feature decoding, and at the 111 end of the decoder, a convolution filter with kernel size = 1×1 used as the network 112 output layer. The entire network will eventually output a segmentation prediction result 113 with the same size as the input image. The predicted value of each pixel represents the 114 likelihood that this pixel belongs to a lung cavity. The overall lung cavity segmentation 115 network is shown in Fig 2, using the Dice coefficient as loss function which is same to 116 original U-Net [12].

118
In this section, we design a multi-model ensemble network, which is combined with a 2D 119 segmentation network and a 3D detection network to sensitively screen the nodule 120 candidates.

122
NestedUNet [22] can accelerate network optimization by using dense convolutional 123 blocks which bridge the semantic gap between the feature maps of the encoder and 124 decoder. However, we found that NestedUNet with depth supervision module is difficult 125 to be optimized and easy to cause gradient explosion while training for pulmonary 126 cavity segmentation in this work. Thus we use the fast version of NestedUNet with the 127 same loss function from the original NestedUNet as our segmentation module, the 128 specific formulas are summarized bellow: where p n is the probability that pixel n is predicted as lung cavity, r n is the true 132 category of pixel n, r n = 1 for the lung cavity and, r n = 0 is regarded as backgound.
where p ∈ {0, 1} (0 for negative and 1 for positive). The classification loss L cls is the 154 cross-entropy loss and the regression loss L reg is the smooth L1 loss function.

156
After a CT scan volume is fed to the aforesaid 2D and 3D pulmonary nodule extraction 157 methods respectively, two lists of pulmonary nodule candidates will be provided, here 158 denoted as L 1 and L 2 . Then we can combine these candidates by using algorithm 1.

159
Firstly, based on NMS algorithm, we remove the parts of candidates whose detection 160 boxes have significant overlaps with others. After that, we delete the parts of candidates 161 from 2D segmentation that have high overlaps with above 3D handled results also based 162 on NMS. The 3D detection method has high accurate location as benefit point however 163 its recall rate is relatively lower than 2D segmentation method. Meanwhile, using 2D DenseNet [24] and SENet [25]. For the center of a given nodule candidate, we extract a 177 3D data cube of size 32 × 32 × 32 that possibly includes pulmonary nodule as input to 178 the three network models, and then get the probabilities p i (i ∈ {1, 2, 3}) of the three 179 classifiers. To further eliminate false positives, we design the strategy described as   b) 3D RPN for Nodule Extraction: As the 3D RPN network is deep, and has more 243 parameters than 2D RPN, the model is easy to overfit on small dataset. To solve this 244 problem, we use similar methods with above task to enlarge the number of training 245 data. In Section we have discussed the significance of the location of tiny nodule in 246 early pulmonary nodule detection. Hence we increase the sampling frequencies of tiny 247 nodules in the training set. Specifically, the sampling frequencies of nodules larger than 248 10 mm and 20 mm are expanded 2 and 4 times higher than other nodules, respectively. 249 The sampling frequency of tiny nodules smaller than 3mm also increases 2 times. Some 250 of negative samples have similar appearances with pulmonary nodules, making them 251 The training procedure of three classification networks are almost identical. We use 268 SGD with the momentum of 0.9 as optimizer to minimize the loss function and update 269 the model parameters. The initial learning rate is 0.001 and decay is 0.9. The batch size 270 is set to be 128. Due to the imbalance problem of positive and negative samples, we 271 finally choose focal loss [26] as loss function. And the experiments proved that the effect 272 of focal loss is better than cross entropy loss.

274
Lung segmentation is very important for analyzing lung related diseases. In this 275 experiment, we compare the performance of our FCNN model with the current methods 276 on dataset [28], including the traditional benchmark method [4], U-Net model [12], 277 R2U-Net [29], and LGAN [30]. 2D Dice score, F1 score and sensitivity were calculated 278 with same settings for each method. The performance comparison is shown in Table 1. 279 Compared with the state of the art methods, our FCNN model has the highest score, 280 with an average F1 score of 0.9836, an accuracy of 0.9965, and an average Dice score of 281 0.9864. Although the traditional method had the highest specificity, it requires a series 282 of thresholds, morphological manipulations, and composition analysis. Compared with 283 this method, our method provides an end-to-end solution, which takes an average of 3.4 284 seconds per scan.

285
In order to visualize the well performance of our proposed architecture, we compare 286 the predicted results of our FCNN on two different CT slices with the traditional 287 Fig 5. Comparison with Traditional Method from F.Liao [4] and Our FCNN. The first column lists two different scans and their corresponding ground truth in second column. In the third column are the segmentation results by F. Liao and in last column are the results by our method. The significant improvement in lung segmentation by using our FCNN model can be 291 observed.

292
We trained the detection model on LUNA16 trainset and evaluated the performance 293 on the validation set of LUNA16. LNDb and ChestCT2019 are also evaluated. In 294 addition, we compared the performance with the DSB and DeepLung models on the 295 three dataset [4,5]. DSB model from Data Science Bowl 2017 pulmonary nodule 296 detection competition and DeepLung from LUNA16 competition. The average recall 297 comparison of the detection models is shown in Table 2.

298
From the above table, we can see that our model ensemble method performs better 299 on each data set than the others. The average recall of our model ensemble method on 300 the LUNA16 is 90.4%, 45.57% on LNDB and 37.01% on ChestCT2019. The 301 performance of the only 3D-RPN used method is poor than our complete method, but it 302 takes the shortest time. In [4], the average recall of the DSB method model trained on 303 the Data Science Bowl 2017 competition data set is 0.8562, but it is designed to neglect 304 the very small nodules during training, the LUNA16 dataset is not suitable.

306
In this paper, we propose a high-performance pulmonary nodule detection system based 307 on stacking of deep convolution networks. We propose a novel full convolution 308 segmentation network based on U-Net for lung segmentation to solve the efficiency 309 problem of the existing pulmonary nodule detection systems. Furthermore, a 3D-RPN 310 detection network and a 2D-NestedUNet segmentation network is stacked to get the 311 high recall and low false positive rate on pulmonary nodule extraction. Finally, a false 312 positive reduction method based on multi-model ensemble is proposed for the further 313 classification of nodule candidates. Extensive experimental results on public available 314 datasets, LUNA16, LNDb and ChestCT2019, demonstrate the superior performance of 315 our CAD system. We believe that our CAD system will be a very powerful tool for 316 early diagnosis of pulmonary cancer.