PT  - JOURNAL ARTICLE
AU  - Falgun H. Chokshi
AU  - Bonggun Shin
AU  - Timothy Lee
AU  - Andrew Lemmon
AU  - Sean Necessary
AU  - Jinho D. Choi
TI  - Natural Language Processing for Classification of Acute, Communicable Findings on Unstructured Head CT Reports: Comparison of Neural Network and Non-Neural Machine Learning Techniques
AID  - 10.1101/173310
DP  - 2017 Jan 01
TA  - bioRxiv
PG  - 173310
4099  - http://biorxiv.org/content/early/2017/08/10/173310.short
4100  - http://biorxiv.org/content/early/2017/08/10/173310.full
AB  - Background and Purpose To evaluate the accuracy of non-neural and neural network models to classify five categories (classes) of acute and communicable findings on unstructured head computed tomography (CT) reports.Materials and Methods Three radiologists annotated 1,400 head CT reports for language indicating the presence or absence of acute communicable findings (hemorrhage, stroke, hydrocephalus, and mass effect). This set was used to train, develop, and evaluate a non-neural classifier, support vector machine (SVM), in comparisons to two neural network models using convolutional neural networks (CNN) and neural attention model (NAM) Inter-rater agreement was computed using kappa statistics. Accuracy, receiver operated curves, and area under the curve were calculated and tabulated. P-values < 0.05 was significant and 95% confidence intervals were computed.Results Radiologist agreement was 86-94% and Cohen’s kappa was 0.667-0.762 (substantial agreement). Accuracies of the CNN and NAM (range 0.90-0.94) were higher than SVM (range 0.88-0.92). NAM showed relatively equal accuracy with CNN for three classes, severity, mass effect, and hydrocephalus, higher accuracy for the acute bleed class, and lower accuracy for the acute stroke class. AUCs of all methods for all classes were above 0.92.ConclusionsNeural network models (CNN & NAM) generally had higher accuracies compared to the non-neural models (SVM) and have a range of accuracies that comparable to the inter-annotator agreement of three neuroradiologists.The NAM method adds ability to hold the algorithm accountable for its classification via heat map generation, thereby adding an auditing feature to this neural network.NLPNatural Language ProcessingCNNConvolutional Neural NetworkNAMNeural Attention ModelHERElectronic Health Record