Summary
The rapidly burgeoning quantity and complexity of publications makes curating and synthesizing information for meta-analyses ever more challenging. Meta-analyses require manual review of abstracts for study inclusion, which is time consuming, and variation among reviewer interpretation of inclusion/exclusion criteria for selecting a paper to be included in a review can impact a study’s outcome. To address these challenges in efficiency and accuracy, we propose and evaluate a machine learning approach to capture the definition of inclusion/exclusion criteria using a machine learning model to automate the selection process. We trained machine learning models on a manually reviewed dataset from a meta-analysis of resilience factors influencing psychopathology development. Then, the trained models were applied to an oncology dataset and evaluated for efficiency and accuracy against trained human reviewers. The results suggest that machine learning models can be used to automate the paper selection process and reduce the abstract review time while maintaining accuracy comparable to trained human reviewers. We propose a novel approach which uses model confidence to propose a subset of abstracts for manual review, thereby increasing the accuracy of the automated review while reducing the total number of abstracts requiring manual review. Furthermore, we delineate how leveraging these models more broadly may facilitate the sharing and synthesis of research expertise across disciplines.
Competing Interest Statement
Dr. Singh has received research support from Stanford's Maternal Child Health Research Institute and Department of Psychiatry, National Institute of Mental Health, National Institute on Aging, Johnson and Johnson, Allergan, Patient-Centered Outcomes Research Institute, and the Brain and Behavior Research Foundation. She is on the advisory board for Sunovion, is a consultant for Limbix, has been a consultant for Google X, and receives royalties from the American Psychiatric Association Publishing. Dr. Ridout receives support from The Permanente Medical Group's Physician Researcher Program. No other authors report any biomedical financial interests or potential conflicts of interest.