SUMMARY
Background Current diagnostic techniques are inadequate for rapid microbial diagnosis and optimal management of patients with suspected sepsis. We assessed the impact of metagenomic sequencing and host response profiling individually and in combination on microbiological diagnosis in these patients.
Methods In this cohort study of 200 consecutive patients with suspected sepsis we evaluated three molecular diagnostic methods with blood specimens: 1) direct bacterial DNA detection and characterization with metagenomic shotgun next generation sequencing and contaminant sequence removal using Bayesian inference; 2) direct viral DNA and RNA enrichment and detection with viral capture sequencing; and 3) transcript-based host response profiling with a previously-defined 18-gene qRT-PCR assay. We then evaluated changes in diagnostic decision-making among three expert physicians in a chart review by unblinding our three molecular test results in a staged fashion.
Findings Metagenomic shotgun sequencing confirmed positive blood culture results in 14 of 26 patients. In 17 of 200 patients, metagenomic sequencing and viral capture sequencing revealed organisms that were 1) not detected by conventional hospital tests within 5 days after presentation, and 2) classified as of probable clinical relevance by physician consensus. Host response profiling led at least two of three physicians to change their diagnostic decisions in 46 of 100 patients. Finally, we report on potential bacterial DNA translocation in 8 patients who were originally classified by physicians as noninfected and show how host response profiling can guide interpretation of metagenomic shotgun sequencing results.
Interpretation The integration of host response profiling, metagenomic shotgun sequencing, and viral capture sequencing synergistically enhances the utility of each of these approaches, and may improve the diagnosis of infections in patients with suspected sepsis.
Funding National Institutes of Health, Chan-Zuckerberg Biohub Microbiome Initiative, and the Thomas C. and Joan M. Merigan Endowment at Stanford University.
Evidence before this study Our PubMed search for articles matching the terms (“metagenomic” OR “cell-free DNA”) and “infect*” in the title/abstract and using the “Human” species filter from inception to September 30, 2019 yielded 463 articles. Many proof-of-concept and validation studies illustrating how metagenomic sequencing can diagnose infections have been previously reviewed. Our search identified only nine studies which applied metagenomic shotgun sequencing to blood specimens, likely because there is a relatively low signal-to-noise ratio with this specimen type in this setting. In a study of 358 febrile sepsis patients, plasma cell-free DNA sequencing detected causative agents missed by standard-of-care testing in 15% of patients, but also detected bacterial organisms adjudicated as commensals in 10% of patients. Recently, a proof-of-concept study used machine learning to integrate metagenomic sequencing and transcriptional host response profiling to differentiate pathogens from commensal organisms in respiratory specimens, albeit with only a small derivation cohort to train host response signatures.
Added value of this study Our 200-patient study assessed the clinical utility of combining both metagenomic sequencing and a previously-defined host response assay to diagnose sepsis. We developed a rigorous chart review approach to measure whether our assays’ results could change a physician’s diagnostic decision-making, without having to commit the assays into patient care. Metagenomic sequencing revealed previously-undetected and clinically relevant organisms in 17 of 200 patients, and host response profiling led at least two of three physician chart reviewers to change their diagnostic classifications in 46 of 100 patients. We also report on potential bacterial DNA bloodstream translocation in 8 of 40 patients who were originally classified by physicians as noninfected and show how host response profiling can guide interpretation of metagenomic shotgun sequencing results. Finally, we present a statistical algorithm for contaminant removal from metagenomic sequencing data using Bayesian inference.
Implications of all the available evidence Current diagnostic techniques are inadequate for rapid microbial diagnosis and optimal management of patients with suspected sepsis. Metagenomic sequencing, which offers the promise of hypothesis-free testing to discover new organisms that would have otherwise been missed, is already being introduced into clinical practice. However, interpretation of results from this powerful approach can be difficult, given that a large fraction of positive results represents reactivated viruses, chronic infections, commensal organisms, and contamination. Host response profiling can serve as an objective adjunct in interpreting ambiguous metagenomic sequencing results. As host response assays are introduced into clinical practice, we suggest that all patients undergoing metagenomic sequencing be simultaneously tested with one of these assays. For now, we urge clinicians to carefully interpret metagenomic sequencing results with the utmost regard for patient safety and antimicrobial stewardship.
INTRODUCTION
The early recognition and diagnosis of severe infection and sepsis is a significant clinical priority. Despite advances in microbial detection methods, clinicians typically rely on presumptive clinical diagnoses and empiric therapy with broad-spectrum antimicrobials, increasing the risks for adverse drug effects1 and the development of antimicrobial resistance. Two emerging approaches, metagenomic sequencing and host response profiling, may each promote the rapid diagnosis of sepsis. Their use in a prospective fashion, and especially in combination, has not been adequately assessed and deserves careful study.
In theory, metagenomic sequencing can identify any microorganism to the species- or strain-level without the need for a prior hypothesis or reliance on cultivation, as long as there are nucleic acids of sufficient abundance and length from the organism(s) in the specimen. Case reports, validation, and interventional studies have highlighted the potential power of this approach2–7. Some methods incorporate microbial enrichment or human depletion steps in order to improve ‘signal to noise’ ratios8,9. For example, viral capture sequencing for vertebrate viruses (VirCapSeq-VERT) is a metagenomic sequencing approach that enriches for all 207 viral taxa known to infect vertebrates (including humans) with sensitivity similar to the real-time polymerase chain reaction assays currently employed in clinical microbiology laboratories10,11.
The mere presence of specific molecular components of an infectious agent in a patient is insufficient however to incriminate the agent as the cause of that patient’s disease12,13. For example, the presence of bacterial nucleic acids in a specimen of blood could be explained by contamination of the specimen with skin bacteria or their DNA during collection14, or even normal low-level translocation of commensal bacteria or their components into the bloodstream during states of health15. Viral sequences may represent latent or clinically-irrelevant viruses in circulating blood cells or their nucleic acids in plasma. Contamination of specimens with microbial nucleic acids from laboratory reagents at the time of specimen processing has been shown to critically affect results in the study of low-microbial biomass samples, such as blood16. Finally, false-positive and -negative results may reflect bioinformatic errors17 and faulty reference databases3,18, or other technical errors. The failure to address these same challenges in the use of other nucleic acid-based testing approaches such as multiplex pathogen PCR panels and C. difficile PCR testing has led to unnecessary antimicrobial treatments, delayed diagnoses, and/or detrimental patient outcomes19–22. The risks of these adverse outcomes are magnified with metagenomic approaches because of their broad range and the ubiquity of microbial nucleic acids.
Assessments of host response to infection offer the possibility of revealing mechanism, inciting factors, and outcome. Although well-established in clinical practice, most traditional analytes, such as acute phase reactants, are non-specific. Host RNA transcript-based profiles can provide evidence of a clinically relevant response with specificity for all infections or broad classes of infectious agents23–27. Thus, these methods offer complementary benefits to methods that only detect microbial signals28. RNA signatures that identify whether a patient is infected and the general type of infection25,29,30 (e.g. bacterial or viral) may be able to provide results in a turnaround time that would allow for initial treatment guidance, since relevant host mRNAs are highly abundant and require relatively little sample preparation. These assays, however, are limited as they generally do not provide species-level information about the causative agent.
We hypothesized that metagenomic sequencing and host response profiling could reveal clinically useful information that current, routine diagnostic tests fail to provide about the potential cause of suspected sepsis, and that their use in combination could prove complementary. Langelier et al. provided the first integration of these two approaches to diagnose lower respiratory tract infections31. In our study, we prospectively enrolled 200 consecutive adult patients who presented to the Emergency Department with suspected sepsis, as defined by a prior sepsis definition32. Next, we applied three molecular approaches with specimens from these 200 adult suspected sepsis patients: 1) metagenomic shotgun next-generation sequencing (mNGS) for bacteria detection in plasma specimens; 2) VirCapSeq-VERT for DNA and RNA virus detection in plasma specimens; and 3) a previously-defined human response-based transcript signature, Integrated Antibiotics Decision Module (IADM)25, to classify bacterial infection, viral infection, and noninfection-based inflammation in whole blood samples. In addition, we developed an open-source gamma-Poisson mixture model-based Bayesian method for distinguishing blood-associated sequences (signal) from reagent-associated, contaminant DNA sequences (noise) in mNGS data. Three physicians with specialty training in infectious diseases performed chart reviews on all patients in a blinded manner and then were provided results from the three diagnostic methods in a staged fashion. We report on the added value of these methods alone and together in generating clinically relevant diagnoses.
METHODS
Subject Enrollment
This study was approved by the Stanford University Administrative Panel on Human Subjects Research (Protocols 32851, 29803, and 29733).
The patient cohort was a prospective, consecutive convenience sample of 200 patients with suspected sepsis. Plasma and PAXgene™ RNA whole blood samples were prospectively collected from adult patients presenting to the Stanford University Hospital Emergency Department (ED) who satisfied all of the following inclusion criteria:
Not pregnant;
Met 2 of 4 SIRS criteria, as defined by Bone et al.32; and
Had suspicion of infection in ED, as determined by triage nurses or other clinicians
Blood samples for this study were collected at the time of venipuncture for standard-of-care bacterial cultures during presentation under a waiver of informed consent granted by the IRB. From this patient sample bank, we then identified 200 consecutive suspected sepsis patients (spanning a 128-day period in 2016) (see Supplementary Attachment 1) who met the following additional criteria:
2.5 mL of whole blood in a PAXgene RNA tube (collected as part of this study protocol) and at least 200 µL of plasma, were available;
blood samples underwent nucleic acid extractions without errors; and
no access restrictions for their electronic medical record
We note that our patient sample banking operations began before the new sepsis-3 definition33 was released in 2016 and relied on a prior definition32. Thus, our enrollment efforts did not include patients that would have otherwise been identified under the expanded sepsis-3 definition.
In addition, we collected 2.5 ml of peripheral blood in a PAXgene RNA tube from each of 10 healthy adult volunteers in the San Francisco Bay Area to serve as controls for host response profiling. Written, informed consent was obtained from each healthy volunteer prior to sampling. Inclusion criteria for these volunteers are available in the Supplementary Methods.
mNGS
DNA was extracted from 200-400 μL of plasma with the QIAamp Circulating Nucleic Acid Kit (QIAGEN). DNA extraction was performed in batches of 24, with 3-4 negative controls per batch, consisting of molecular-grade water, to monitor environmental and reagent contamination during sample processing. Libraries were prepared with the KAPA HyperPrep Kit (Roche) at the High-Throughput Sequencing and Genotyping Unit at the University of Illinois at Urbana-Champaign, and sequenced on the HiSeq 4000 (Illumina) with 2×150 nucleotide paired-end reads.
In a pilot experiment, we sequenced DNA from a plasma sample from each of 15 patients with a positive bacterial blood culture, at a depth of 40M-60M reads/sample, and 4 negative controls at a depth of 2-6M reads/sample. We then sequenced plasma samples from the other 185 patients each to a depth of 10M-52M reads using unique dual-indexed barcodes, alongside 36 negative control samples sequenced to a depth of 3M-6M reads.
After adapter sequence removal, quality trimming, and human genome sequence subtraction, Kraken34 was run on non-human reads with a conservative alignment threshold of 0.3. Bacterial reads classified to the species-level were further analyzed with phyloseq35.
Exploratory analysis using PCA on rank-normalized sequence reads showed possible batch-effects which may have contributed to variation in sample sequence composition (figure S1). Two distinct clusters were visualized when samples were grouped into sets based on their extraction batches: Set 1, consisting of all samples from extraction batches 1, 2, 3, 4, 11, and 12; and Set 2, which consisted of all samples from extraction batches 5, 6, 7, 8, 9, and 10. The Pilot Set, consisting of all the pilot experiment samples, clustered with Set 1. Closer examination of taxa in the negative controls of Set 1 and Set 2 revealed distinct contamination signatures, with numerous high-abundance taxa unique to each set (figure S2). We hypothesized that differences in manufacturing lots of the nucleic acid extraction kits may have caused the variation in sample sequence composition, as Glassing et al. previously reported36. Further details on mNGS sample preparation, bioinformatics, and exploratory analysis are provided in supplementary methods.
To distinguish blood-associated DNA sequences from contaminant sequences in plasma samples, we developed a Bayesian statistical method that leverages data from negative control samples. We ran the contaminant removal algorithm separately on the three sets of samples: Set 1, Set 2, and the Pilot Set. We analyzed the Pilot Set separately, even though it behaved similarly to Set 1, because the two sets were extracted by separate technicians and sequenced using different barcode adapters several months apart. A detailed description for our contaminant removal algorithm is provided in appendix S1, and an open-source R package of this method is available at https://github.com/PratheepaJ/BARBI.
VirCapSeq-VERT
Nucleic acid was extracted from 150 μl of plasma using the NUCLISENS easyMAG system (bioMerieux). Sequencing libraries were prepared with the KAPA HyperPrep kit (Roche) following reverse transcription (Superscript III; Thermo Fisher) and second-strand synthesis (Klenow polymerase; New England Biolabs). Libraries were labeled with custom unique dual barcode adapters (Integrated DNA Technologies) and pooled for VirCapSeq-VERT capture hybridization (25-32 samples per pool). The VirCapSeq-VERT enriched library pools were sequenced on a HiSeq 2500 sequence analyzer (Illumina), generating 1 x 100 nucleotide single end reads.
Sequence reads were demultiplexed, Q30-filtered and assessed by RRINSEQ (v0.20.2) prior to host sequence subtraction and de novo assembly (MIRA v4.0). Contigs and unique singletons were subjected to homology search using MegaBLAST and the GenBank nucleotide database. Sequences not assigned at the nucleotide level were screened by BLASTx to detect divergent or potentially new viruses. Based on the BLAST results, contigs and singletons were assigned at family, genus, species and GenBank accession number level to identify the most closely related GenBank entries.
Each sample pool included a negative control consisting of Salmon nucleic acid that was processed alongside the human plasma samples. Raw read counts were normalized (reads per 10,000 host subtracted total reads) and a positive score assigned to specimens with a result >0.2 and for which these reads did not represent a read pile-up in one position but were distributed to three or more genome regions. We did not report on viruses of the family Anelloviridae and GB viruses as they have no established clinical significance in humans37,38. Additional details on VirCapSeq-VERT sequencing and bioinformatics processing are available in supplementary methods.
Host RNA transcript profiling
We tested samples from 193 patients and 10 healthy adult volunteers with a previously described 18-gene host-response assay consisting of 1) an 11-gene set to distinguish noninfection- and infection-associated SIRS, the Sepsis MetaScore (SMS)24; and 2) a 7-gene set to distinguish bacterial and viral infections, the ‘bacterial-viral metascore’ (BVS)25. The Stanford Functional Genomics Facility extracted RNA from PAXgene RNA tubes using the QIAcube system (Qiagen) according to the manufacturer’s recommendations, and then performed qRT-PCR for specific human transcripts in triplicate using commercial TaqMan assays on the Biomark HD platform (Fluidigm). Samples from seven patients were not profiled because of failure of PCR amplification. SMS and bacterial-viral scores were calculated as previously described25.
Since this was the first use of qRT-PCR to measure target mRNAs, we needed to re-establish SMS and BVS cutoffs for the data generated in this study. First, physicians with subspecialty training in infectious diseases (not the three physicians of the main chart review) conducted a ‘host response calibration chart review’ to establish baseline classifications of infection status and type for the 193 patients with host response results. Each patient’s medical records were reviewed by two physicians who were blinded to mNGS, VirCapSeq-VERT, and host response profiling results. SMS and BVS score cutoffs were then re-established using the results from the ‘derivation cohort’ of 93 patients who were adjudicated as noninfected, or as having a bacterial or viral infection by physicians with evidence from standard-of-care microbiological tests. Cutoffs were set to incorporate 95% sensitivity for bacterial infections. With these score cutoffs, host response classifications of ‘bacterial,’ ‘viral,’ or ‘noninfected’ were generated for all 193 patients. In the main chart review, physicians were presented with plots of host response results incorporating score cutoffs for only the 100 patients in the ‘test cohort’. The host response calibration chart review questions and results are available in supplementary methods and supplementary attachment 1, respectively.
Main Chart Review
We recruited three physicians with subspecialty training in infectious diseases, to perform a retrospective chart review on the 200 patients in a blinded manner. They were asked to make classifications on infection status and clinical relevance of mNGS and VirCapSeq-VERT results, in a staged fashion: 1) with only medical charts; 2) with the addition of mNGS and VirCapSeq-VERT results; and 3) with the further addition of host response results. The results of chart review are summarized in figure 1, and details are provided in supplementary methods, appendix S2, and appendix S3.
We applied three diagnostic approaches to our cohort of 200 adult patients with suspected sepsis: 1) direct bacterial DNA detection and characterization with plasma metagenomic ‘shotgun’ next generation sequencing (mNGS) and contaminant sequence identification using Bayesian inference; 2) direct viral DNA and RNA enrichment and detection with plasma viral capture sequencing (VirCapSeq-VERT); and 3) transcript-based host response profiling with a previously-defined 18-gene qRT-PCR assay on whole blood. Additionally, two separate chart reviews were performed. First, a ‘Host Response Calibration Chart Review’ established baseline diagnoses of all patients for the sole purpose of calibrating host response cutoffs. Second, a ‘Main Chart Review’ evaluated changes in diagnostic decision-making among three expert physicians in a chart review by unblinding our three molecular test results in a staged fashion.
RESULTS
Patient population
We recruited 200 consecutive patients in the ED with suspected sepsis; applied mNGS, VirCapSeq-VERT, and host response profiling on blood specimens of each patient; and evaluated patient clinical records in two separate physician chart reviews, as depicted in figure 1.
Patient demographics are listed in table 1. The clinical syndromes at presentation were diverse and included fever without localizing findings (32% of patients), as well as syndromes involving the respiratory (21.5%) and genitourinary (9.5%) tracts, and intra-abdominal sites (16.5%). While these patients were enrolled because they met SIRS criteria and were suspected at the time of presentation by triage nurses in the ED of having sepsis, physicians classified 16 patients (8%) as not infected during the main chart review while blinded to mNGS, VirCapSeq-VERT, and host response profiling results. The remaining patients were classified as having bacterial infections (69 patients, 34.5%), viral infections (11 patients, 5.5%), fungal infection or coinfection (4 patients, 1%), or probable or unsure status (100 patients, 50%) (figure 2). Changes in physician classifications made after considering mNGS, VirCapSeq-VERT, and host response results are also summarized in figure 2.
At each phase of our main chart review, patients were either assigned with high confidence to one of four diagnostic categories by a panel of three physicians or classified to have only a probable (e.g. probably bacterial, probably noninfected) or unsure diagnosis. Physicians did not evaluate host response scores from seven patients who had host response assay fail due to amplification errors, and 93 patients who had host response scores used to set cutoffs. For Phase III, the same classification from Phase II was kept for patients who did not have host response scores for evaluation.
Comparison of mNGS and VirCapSeq-VERT with standard-of-care microbiology
To distinguish signal from noise and remove contaminant sequences from plasma sequence data, we developed a gamma-Poisson mixture model-based Bayesian inference method. Using the 40 negative control samples, this method identified the vast majority of taxa in our dataset as contaminants (figure S3). Subsequent analyses of mNGS output were performed on contaminant-filtered data.
Bacterial sequences were identified by mNGS in plasma matching those of the species cultivated from blood collected at the same time, from the same subject in 14 of 26 patients with positive blood cultures (table 2). Interestingly, mNGS results were also concordant with the positive results of urine or sputum cultures performed within 1 day of presentation in 3 of 24 patients with negative blood cultures (table 2). To test whether sequencing depth might explain low sensitivity, we selected plasma samples from 7 patients with positive blood, urine, wound, or bronchoalveolar lavage cultures but negative mNGS results and acquired an additional 65-262M reads per sample. With these additional data, 2-111 additional sequencing reads matching the species of the isolated organism(s) were recovered in 6 of the 7 samples (table S1).
VirCapSeq-VERT high-throughput sequencing was performed on 199 of the 200 available plasma samples. One of the 200 samples failed to yield sufficient nucleic acid for analysis despite repeated extraction attempts. An average of 12 million raw reads were obtained for each of the 199 samples using this approach. In comparison to standard-of-care PCR testing on plasma, VirCapSeq-VERT confirmed the presence of cytomegalovirus DNA in 2 of 2 subjects (table 2). VirCapSeq-VERT analysis of plasma did not provide evidence of viruses that were subsequently identified by PCR tests on respiratory and stool samples in 8 patients as well as a heterophile antibody (Monospot) Epstein-Barr Virus test on 1 patient performed within 1 day of presentation (table 2, supplementary attachment 1). However, respiratory or stool samples were not tested using VirCapSeq-VERT and there were no independent molecular or culture data indicative of viremia.
To address whether organisms detected by mNGS or VirCapSeq-VERT were likely etiologic agents for the clinical presentation, our three expert physicians independently evaluated the mNGS and VirCapSeq-VERT findings in the main chart review. Of the 40 patients with organisms detected by mNGS in plasma from the day of presentation that were not identified with standard-of-care microbiological testing performed within 5 days of presentation, organisms in 14 patients were classified by physician consensus as either ‘probably clinically relevant’ or ‘clinically relevant’ (figure 3 and table 3). The addition of mNGS results led physicians to change their classification for the presence of bacterial infection in just six of these 14 patients by consensus (figure 3). The remaining eight patients were already established as known bacterial infection patients by physician consensus before mNGS results were revealed (figure 3).
Tables and Venn diagrams illustrate the number of patients with clinically relevant mNGS and VirCapSeq-VERT results which reveal the etiologies of patient presentations and change diagnostic decision-making. Patients were grouped by those who had positive (A) mNGS and (B) VirCapSeq-VERT results which 1) were not detected by standard-of-care microbiology performed within 5 days after presentation, 2) were classified as ‘clinically relevant’ or probably ‘clinically relevant’ to the patient’s presentation by physician consensus while blinded to host response results during the main chart review, and 3) led physicians to have a consensus change in classification for presence of bacterial/viral infection by at least 1 point on our 5-point Likert scale. Patients were segmented into five groups using consensus classifications made during the main chart review when physicians were blinded to mNGS, VirCapSeq, and host response results. Details for all patients with positive mNGS and VirCapSeq-VERT results which were not previously detected by standard-of-care microbiological testing are presented in Table S2 and S4, respectively.
The potential bacterial etiologies found only by mNGS included Streptococcus mitis (Pt_154), Borrelia hermsii (Pt_083), Leptospira interrogans (Pt_163), and Haemophilus influenzae (Pt_194) (table 3). In five patients with positive blood cultures (Pt_020, Pt_037, Pt_092, Pt_137, and Pt_145), mNGS uncovered additional organisms that were not found in the blood cultures (table 3). mNGS results and clinical details for all 40 patients with mNGS organisms not detected by standard-of-care microbiological testing are presented in table S2. We note that in 16 of these 40 patients mNGS revealed only organism(s) that represented possible contaminants (and/or misalignment errors). These organisms remained after contaminant removal because they were not known to be typically associated with humans or had relatively few reads recovered (table S2).
Of the 27 patients with viruses detected by VirCapSeq-VERT in plasma from the day of presentation that were not identified with standard-of-care microbiological tests, three patients had viruses which were classified by physician consensus as either ‘probably clinically relevant’ or ‘clinically relevant’ to the patient’s presentation (figure 3 and table S3, with clinical details for each patient presented in table S4). Two of these patients had Coxsackievirus sequences, and one had probable Epstein-Barr Virus infection. VirCapSeq-VERT results from all three patients led to a change in the physicians’ consensus classification for the presence of viral infection. Although they were not classified as the etiologies of the patient presentation, VirCapSeq-VERT demonstrated utility in detecting potential chronic viral infection or viral reactivation (supported by prior documented lab findings or signs and symptoms) in 18 patients with human herpesvirus 6, hepatitis C virus, hepatitis B virus, BK virus, Epstein-Barr virus, or Trichodysplasia spinulosa-associated polyomavirus. The remaining virus sequences had uncertain clinical relevance (tables S3 and S4).
Details on four patients for whom physician classifications of infection status and type were most altered by the results of mNGS and VirCapSeq-VERT are presented in table 4. These patients were found to be infected with coxsackieviruses, Borrelia hermsii, and Leptospira interrogans, which all defied clinical suspicion by the treating physicians. The patient with the greatest change in physician classification was Pt_163, a febrile male with headache and diarrhea upon return from travel to South Asia who was believed to have had a viral infection but was found to have Leptospira interrogans sequences in plasma by mNGS.
Impact of Host Response Profiling Results on Physician Classifications
To evaluate the impact of host mRNA response signatures on physician classifications of patients, we applied the previously-established Integrated Antibiotics Decision Module (IADM)25 on 193 patient samples. The IADM incorporates the Sepsis MetaScore (SMS) which distinguishes noninfection- and infection-associated SIRS, and the Bacterial/Viral metaScore (BVS) which distinguishes bacterial and viral infections (figure 4A). Since this study was the first use of qRT-PCR to measure target mRNAs of the SMS and BVS, new cutoffs were set using the results from the ‘derivation cohort’ of 93 patients adjudicated by physicians in a separate chart review as either non-infected, or having bacterial or viral infection (figure 4B). With these new cutoffs, host response classifications of ‘bacterial,’ ‘viral,’ or ‘noninfected’ for all 193 patients were generated and compared against physician adjudications (figure 4C-D).
(A) Schematic for two numeric scores of host response assay. The Sepsis MetaScore (SMS) distinguishes noninfection- and infection-associated SIRS, and the Bacterial/Viral metaScore (BVS) distinguishes bacterial and viral infections. (B) SMS and BVS score cutoffs were re-established using the results from the ‘derivation cohort’ of 93 of 193 patients for whom physicians adjudicated as noninfected, bacterial, or viral in a separate ‘host response calibration chart review.’ With these score cutoffs, host response classifications of ‘bacterial,’ ‘viral,’ or ‘noninfected’ were generated for all 193 patients. In the main chart review, physicians were presented with plotted host response results with score cutoffs (C) for only the 100 patients in the ‘test cohort’ to interpret. (C) Distribution of scores and cutoffs for the host response assay. A higher SMS indicates a higher chance of infection over noninfection, and a higher BVS indicates a higher chance of viral infection over bacterial infection. (D) Confusion matrix for the host response assay vs. adjudicated physician classifications. The following characteristics were calculated from derivation cohort patients with total n = 93: bacterial infection sensitivity, 93.9%; bacterial infection specificity, 73.2%; viral infection sensitivity, 44.4%; viral infection specificity, 93.8%; noninfected sensitivity, 31.3%; noninfected specificity, 94.8%.
Among derivation cohort patients, bacterial, viral, and noninfection detection sensitivities were 93.8%, 45.5%, and 43.8% respectively. Bacterial, viral, and noninfection detection specificities were 70.4%, 97.5%, and 94.7%, respectively. The IADM distinguished noninfected and infected patients with an area under curve (AUC) of 0.73 (95% confidence interval [CI] 0.68-0.79), and bacterial from viral infections with an AUC of 0.89 (95% CI 0.85–0.93) (figure S4) using receiver operating characteristic (ROC) analysis.
We then examined the impact of host response profiling results on physician diagnostic decision-making of the 100 test cohort patients who were not used in setting host response score cutoffs. In 46 patients (46%), the addition of host response profiling results led at least two of three physicians to change their classification of infection status and type (figure 5A). We also asked physicians to classify the clinical relevance of mNGS and VirCapSeq-VERT organisms first using medical charts only and then with the addition of host response results. Ten patients had at least one physician change their classification of clinical relevance of an organism revealed by mNGS or VirCapSeq-VERT upon receiving host response scores (figure 5B).
In our main chart review, physicians reviewed host response results of the 100 patients in the validation cohort. (A) Table and Venn diagrams illustrate the number of patients for whom the introduction of host response results led physicians to change their classification(s) of infections status and type by at least one or two points in our main chart review. These classifications included whether the physicians believed the patient had 1) an infection, 2) a bacterial infection, 3) a viral infection, 4) a fungal infection, or 5) a parasitic infection. Response options were on a five-point scale (No-Probably No-Unsure-Probably Yes-Yes). (B) Twenty-five of the 100 test cohort patients had positive mNGS or VirCapSeq-VERT results. After reviewing host response results, ten of the 25 patients had physicians change their classification of the mNGS and VirCapSeq-VERT organism’s clinical relevance to the patient’s presentation. Physician classifications of the ten patients’ mNGS and VirCapSeq-VERT organisms before and after the introduction of host response results are presented in the bottom table.
Possible bacterial DNA bloodstream translocation in patients originally classified as noninfected
In eight of the 50 patients originally classified by physician consensus as probably noninfected or noninfected, mNGS detected sequences in plasma from typical commensal organisms (table S5). Physicians noted pre-existing mucosal membrane disturbances in five of these eight patients, thus raising the possibility of bacterial DNA translocation from heavily colonized mucosal sites. For example, Pt_070, who had high abundances of sequences from more than 20 oral cavity-associated organisms in plasma, had documented gingivitis and hemoptysis. All eight patients improved after their ED visit, six of whom were not prescribed antibiotics. Host response results could have been useful to physicians for interpreting ambiguous mNGS results from these patients. However, most of these patients were in the ‘derivation cohort’ used for setting host response cutoffs, and thus did not have their host response results assessed in the main chart review (figure 4B). Nonetheless, host response profiling predicted that five of the eight patients were not infected (table S5).
Data from mNGS, VirCapSeq-VERT, host response, and physician chart reviews for all 200 patients are provided in supplementary attachment 1.
Discussion
Diagnosing infections in patients with suspected sepsis is challenging, particularly in those with multiple co-morbidities. We applied two broad-range sequencing approaches, mNGS and VirCapSeq-VERT, as well as host response profiling to a prospectively-sampled cohort of 200 adults with suspected sepsis who were enrolled in an Emergency Department. The consecutive convenience sample reflects real-world patient heterogeneity in a tertiary care hospital. We evaluated diagnostic decision-making by three infectious disease physicians as they received information from the electronic medical record, the two sequencing-based methods, and host response profiling in a staged fashion. Our results show that the sequencing methods can detect clinically relevant organisms that were missed by routine microbiological diagnostic methods, as well as other organisms that were not deemed clinically relevant. In addition, we demonstrated the potential for host response profiling to influence diagnostic decision-making and help interpret metagenomic sequencing results.
One of the most important features of unbiased, ‘shotgun’ metagenomic sequencing is that it is hypothesis-free, allowing simultaneous detection of thousands of organisms, including those difficult-to-culture. Seventeen of the 200 patients had clinically relevant organisms detected by mNGS and VirCapSeq-VERT that were not detected by standard-of-care microbiology within five days after presentation. Results from nine of these 17 patients led physicians to change their classifications of infection status and type. For example, patient Pt_083 presented with fever after travel to the Sierra mountains and was presumed to have a urinary tract infection by treating physicians. However, this patient was determined by mNGS to have tick-borne relapsing fever due to Borrelia hermsii. Our positivity rate was comparable to other clinical metagenomics studies as reviewed by others39,40. For example, in a study of 204 meningitis and encephalitis patients diagnoses in 13 of them were made solely by metagenomic sequencing with CSF samples, with an impact on patient management in 7 of the 1341. In a study of cell-free plasma in 358 febrile sepsis patients, 15% of patients had probable causal pathogens detected solely by metagenomic sequencing5. It should be noted that metagenomic sequencing can provide a 53-hour turnaround time5, which is shorter than standard-of-care tests in some situations.
Contaminant sequence identification and computational removal represents one of the greatest barriers to expanding the clinical application of metagenomic sequencing, especially in specimens with low microbial biomass such as blood. The gamma-Poisson mixture model-based Bayesian inference approach that we have introduced here offers an important advance in addressing this challenge. In our implementation, we assumed that DNA sequences in plasma included those of contaminants. We then inferred the true ‘intensity’ of DNA sequences in a plasma sample that might be attributed to ‘true’ blood-associated nucleic acids. This method adds to others available to researchers for contaminant removal4,36,37. For example, for studies with fewer than three negative control samples, simply subtracting species based on their presence or abundance in negative controls42 may be most appropriate. While the decontam43 method is not well-suited for our data because it assumes that samples have a relatively higher biomass than controls, and that each taxon is either a contaminant or ‘true’ but not both, it can be very helpful for 16S rRNA gene amplicon sequence data from other kinds of samples.
As metagenomic sequencing enters clinical practice, it is important to recognize the potential of this powerful approach to reveal true signals, as well as clinically-irrelevant sequences which can occur because of translocation of microbial nucleic acids from heavily colonized body sites, reactivation of latent viruses, or contamination of laboratory reagents or specimen collection devices. Virus sequences from 18 of 27 patients with positive VirCapSeq-VERT results were associated with chronic infections or viral reactivations that were not clinically relevant to the patient’s presentation. Additionally, eight of 50 patients who were originally classified as noninfected or probably noninfected had bacterial organisms detected by plasma mNGS. Five of these eight patients improved without antibiotics. Clinicians are accustomed to the importance of clinical-pathological correlations for establishing the relevance of laboratory findings. With the advent of sensitive molecular diagnostic technologies, this challenge will only grow. Indeed, Blauwkamp et al. detected organisms adjudicated as ‘commensal’ in 36 of 358 febrile sepsis patients (10.0%) and as ‘viral reactivation’ in 10 of 358 patients (2.8%) with metagenomic sequencing of cell-free plasma DNA5. They suggested that sequence abundance and overall clinical picture should be considered while assessing clinical relevance of metagenomics results.
Our data illustrate the utility of transcriptional host response signatures, as an objective adjunct in guiding the interpretation of mNGS results and avoiding misdiagnosis and unnecessary treatment. Our results add to those of Langelier et al., who combined host response and metagenomic sequencing to diagnose lower respiratory tract infections31 using a different approach from ours. Their study included a sophisticated machine-learning-based integration of the complementary approaches, using a training set of 20 patients to generate signatures which identified infectious etiologies vs. commensal organisms in respiratory metagenomic sequencing results. In our study, host response signatures for identifying bacterial vs. viral vs. noninfected patients were previously trained on datasets from over 2,000 patients25 representing a wide diversity of infectious etiologies. Additionally, we focused on measuring the clinical utility of having physician chart reviewers integrate metagenomic sequencing and host response profiling results into their clinical decision-making.
The major limitation of our mNGS protocol was suboptimal sensitivity, which was explained in part by the choice of sequencing depth. Low numbers of patients with known systemic viral infections limited our ability to assess the performance characteristics of VirCapSeq-VERT. Bioinformatic errors17 from contaminant, misannotated, or missing genomes in microbial databases, and other technical limitations such as index-hopping44, may have led to false-negative and false-positive findings. Recent efforts to curate databases45, manufacture contaminant-free extraction kits, and enrich for microbial sequences6,8 are steps in the right direction to prepare metagenomic sequencing for routine clinical use, but much work remains to be done.
Our bacterial contaminant sequence identification method did not subtract all contaminant sequences in our mNGS dataset. Increasing the number of negative control samples in every extraction batch could aid in profiling the large diversity of contaminating taxa and thus enhance contaminant sequence removal. Furthermore, our negative control samples were only suited for identifying extraction reagent contaminants. We were not positioned to account for skin-associated contaminants or spurious sample-to-sample cross-contaminants.
The host response profiling assay classified many viral and noninfected patients as bacterial. One possible reason for these misclassifications was the strict dichotomous cutoffs that we used to distinguish infected vs. noninfected cases, and viral vs. bacterial infections. Reporting results with numeric values rather than dichotomous cutoffs will allow better weighting of these scores in patient assessments. Another reason for the misclassifications was the need to re-establish host response score cutoffs for this study’s qRT-PCR platform and the small number of known viral patients with which to do so. Further work is needed to establish and lock cutoffs, validate on additional patient populations, and quantify test characteristics.
We believe that host response profiling and shotgun sequencing will soon achieve turnaround times of less than 90 minutes and 24 hours, respectively. In our chart review, physicians had access to the patients’ full medical chart histories, including test results that only became available several days after presentation. If it were possible to limit the review to the first 90 minutes of each case history, we expect that there would have been greater changes in diagnostic decision-making across the stages of our chart review.
A central limitation in evaluating new diagnostic tools is the lack of a gold standard. We did our best to address this using expert physicians in a staged chart review. Fundamentally, it is impossible to determine whether the changes in patient classification were correct. However, the measurement of a diagnostic tool’s ability to change clinical decision-making, rather than just a comparison of its results to standard-of-care testing, is a valuable component of establishing clinical utility. An important secondary finding was that clinicians had varying levels of trust in these new diagnostic tools. In conclusion, our proof-of-concept study on a consecutive, prospectively-sampled patient cohort suggests that integrating host response profiling with metagenomic sequencing may synergistically enhance the utility of each assay, and ultimately, the diagnosis of patients with suspected sepsis.
Conflict of Interest Disclosures
Dr. Cheng, Dr. Thair, and Dr. Sweeney are employees of Inflammatix. Dr. Strouts is an employee of Cepheid. Dr Dalai is an employee of Karius. Dr. Lipkin is an advisor to Pathogenica. Dr Khatri is an advisor to Inflammatix. Dr Relman is an advisor to Arc Bio and Karius.
Acknowledgements
We thank Ian Brown, Patrice Callagy, Cheryl Bucsit, and Adele Araya of the Stanford Emergency Department for facilitating the collection of samples for this project. We thank members of the Relman Lab, and in particular, Stephen J. Popper, Eitan Yaffe, Christine L. Sun, Daniela S. Goltsman, and Natalie Campen for valuable advice, feedback, and general assistance. We also thank members of the Lipkin Lab, Joel A. Garcia, Nishit P. Bhuva, and Lokendrasingh Chauhan for technical assistance, and Bohyun Lee and Komal Jain for bioinformatics support, Kelly Murphy (Stanford Emergency Department) for advice and assistance, and John Coller and Xuhuai Ji at the Stanford Functional Genomics Facility for their technical assistance with the Fluidigm assay. We thank Alvaro Hernandez and Chris Wright at the High-Throughput Sequencing and Genotyping Unit at the University of Illinois at Urbana-Champaign for their assistance and support with DNA library preparation and sequencing. This work was supported by NIH U19 AI109761 (D.A.R., W.I.L.), Chan-Zuckerberg Biohub Microbiome Initiative (D.A.R.), and the Thomas C. and Joan M. Merigan Endowment at Stanford University (D.A.R.).