Abstract
Objectives Current antibiotic treatment guidelines on when to consider 3rd generation cephalosporin resistant Enterobacteriaceae (3GC-R EB) as a cause of infection have low specificity, thereby increasing unnecessary carbapenem use. Therefore, we aimed to develop new diagnostic scoring systems to direct initial carbapenem treatment to patients at risk of 3GC-R EB bacteraemia.
Methods A retrospective nested case-control study was performed that included patients ≥18 years from 8 Dutch hospitals in whom blood cultures were obtained and intravenous antibiotics were initiated. Patients with 3GC-R EB bacteraemia were each matched to four control infection episodes within the same hospital, based on blood culture date and onset location (community or hospital). Starting from 32 commonly described clinical risk factors available at infection onset, selection strategies were used to derive scoring systems for the probability of community‐ and hospital-onset 3GC-R EB bacteraemia.
Results Among 22,506 community-onset and 8,110 hospital-onset infections, respectively 90 (0.4%) and 82 (1.0%) were 3GC-R EB bacteraemias. As control populations, 360 community-onset and 328 hospital-onset infection episodes were included. The derived community-onset and hospital-onset scoring system consisted of 6 and 9 predictors, respectively, and both showed good discrimination with c-statistics of 0.807 and 0.842. Cutoffs for the scores could be chosen such that ~20% of patients would be eligible for empirical carbapenem treatment, which would capture ~70% of those with 3GC-R EB bacteraemia.
Conclusions These prediction rules for 3GC-R EB bacteraemia, specifically geared towards the initiation of empiric antibiotic treatment, may improve the balance between inappropriate antibiotics and carbapenem overuse.
Introduction
As a consequence of the emergence of infections caused 3rd generation cephalosporin (3GC) resistant Enterobacteriaceae (3GC-R EB; in this manuscript used synonymously with extended-spectrum β-lactamase (ESBL) producing Enterobacteriaceae), physicians are increasingly faced with the question which patients need empiric antibiotic treatment covering these pathogens. Hence, patients and physicians might benefit from prediction rules for 3GC-R EB. For instance, some studies yielded risk factors for carriage of ESBL-producing Enterobacteriaceae at hospital admission [1–4]. Yet, although relevant for infection control purposes, that knowledge may not accurately predict etiologic causes of infection. Other studies aimed to distinguish ESBL‐ or carbapenemase-producing Enterobacteriaceae as a cause of infection, but only included bacteraemic patients [5–7]. Applicability of these models to choosing empiric treatment is unknown, as among patients with suspected bacterial infection, microbiologically documented Enterobacteriaceae infection only comprises a subset of all possible aetiologies.
Current Dutch empiric treatment guidelines designate patients at risk of infection caused by 3GC-R EB based on prior colonization or infection with 3GC-R EB or on prior exposure to cephalosporins or fluoroquinolones, as these were identified as risk factors in patients with bacteraemia caused by these pathogens [8]. Yet, when evaluated in a cohort of patients needing empiric antibiotic treatment, full implementation would lead to considerable carbapenem overuse, with a low sensitivity to actually detect those patients with bacteraemia caused by 3GC-R EB [9]. Therefore, we aimed to develop prediction rules to identify, among patients needing intravenous empiric antibiotic therapy, those having an infection caused by 3GC-R EB.
Methods
Setting and patients
This was a retrospective nested case-control study involving 8 hospitals, of which 3 university hospitals, in the Netherlands. Between January 1st 2008 and December 31st 2010, we included all consecutive patients of 18 years of age or older in whom a blood culture was obtained and intravenous broad-spectrum β-lactam antibiotics (i.e. not penicillin or flucloxacillin), aminoglycosides, and/or fluoroquinolones were started on the day of the blood culture or the day after, irrespective of duration. Patients receiving any of the eligible antibiotics on the day of blood culture obtainment were excluded if these had been initiated prior to this day (see Supplementary Table 2 for illustrating examples). In addition, patients with 3GC-R EB bacteraemia in the year prior were excluded, as it was assumed that treating physicians would always provide therapy aimed at these organisms in case of renewed infection. Patients could be included more than once, if a subsequent episode complied with in‐ and exclusion criteria. Additional information on hospital characteristics, study periods, and databases used in each of the hospitals is provided in Supplementary Table 1.
Infection episodes were separated into two cohorts: the community-onset cohort comprised episodes in which the first blood culture was collected during the first three calendar days of hospitalization, and the hospital-onset cohort consisted of episodes in which blood cultures were obtained later during hospitalization.
The causative pathogen of each episode was based on the results of blood cultures obtained during the onset days (i.e. the first day if antibiotics were started on the same day as blood culture obtainment and the first two days if antibiotics were started the day after). Cases were defined as patients with 3GC-R EB bacteraemia (see Supplementary Table 1 for definition of 3GC resistance in each of the hospitals). We aimed to include approximately 100 cases for both cohorts, in order to be able to construct an initial logistic regression model with 10 variables, based on the 10 events per variable recommendation [10].
The control population was defined as all other infection episodes, including non-bacteraemic episodes and episodes with blood cultures yielding non-resistant Enterobacteriaceae, other bacteria or fungi. For efficiency reasons, this population was not analysed in its entirety. Instead, four controls were matched to each case, a ratio chosen because of minimal gains in statistical power with more controls [11]. Controls were matched on hospital, being in the community or hospital-onset cohort, and being closest in time to the blood culture day of the case episode.
Due to its retrospective nature, the Dutch Medical Research Involving Subjects Act did not apply to this study. In each of the participating hospitals, applicable local guidelines for non-interventional studies were followed. In accordance with Dutch regulations, informed consent was waived for the study. Reporting of this study was in accordance with the TRIPOD Statement [12].
Data collection
All selected cases and controls were subjected to chart review to obtain information that was considered available at the moment that the initial antibiotics were prescribed (referred to as infection onset). Blinding for the outcome during chart review was not considered feasible. Please refer to Supplementary Table 3 for an overview of all collected variables.
Statistical analysis
Two separate prediction models were constructed, one for community-onset and one for hospital-onset infections. Data analyses were performed in R (version 3.2.2) [13], including packages mice 2.25 [14], rms 4.5-0 [15], pROC 1.8 [16], and xtable 1.8-2 [17]. Descriptive analyses of predictors were based on non-missing data only. Some variables were aggregated because of high correlation, low prevalence, and/or similar associations with the outcome (indicated in Table 2). Additionally, the number of categories for suspected sources was reduced to four by combining categories with low frequencies into a single remaining group (original categories in Supplementary Table 3), and categories for antibiotic use were created based on prevalence and assumed predictive power for 3GC-R EB infection. Twenty imputed datasets were created to deal with missing values (see Supplementary Material for exact procedures).
For both community-onset and hospital-onset infections, the models in Table 1 were constructed. Continuous predictors were initially introduced into models with restricted cubic spline functions with three knots to allow for non-linear associations. In the final models, we evaluated if simplification to a linear predictor was possible. Performances of the final (simplified) models and sensitivity analyses were compared by means of the Akaike’s Information Criterion (AIC). The final simplified models were additionally compared to two basic models, the prior identification model and two-predictor model (Table 1).
Regression coefficients of the final simplified models were pooled over imputation datasets by means of Rubin’s rules and shrunk according to model optimism (see description further on). Also, intercepts of the models were adjusted for the sampling fraction of the controls, and controls were weighted by the inverse of the sampling fraction, as previously described [18]. Calibration of the predicted and observed probabilities was visually inspected for separate imputation datasets. Discrimination was assessed by areas under the curves for receiver operating characteristic curves (referred to as C-statistic), averaged over the imputation datasets. Sensitivity, specificity and positive and negative predictive values, and prevalence (i.e. fraction of the population classified as at risk of 3GC-R EB bacteraemia) were calculated for different cutoffs of the predicted risk, again averaged over the imputation datasets. A simplified score was created by multiplying the regression coefficients with a constant, followed by rounding to easy-to-use values. Performance of this score was determined similarly.
Estimation of model optimism
For the final simplified models, optimism was estimated by creating 2000 bootstrap samples, creating a new prediction model for each of these samples, and comparing the model’s performance in the original and bootstrapped data. Optimism results from the fact that models are developed on a population sample and suffer from overfitting, which jeopardizes generalizability to the total population [19]. By means of bootstrapping, the expected effects when applying the model within the entire population are mimicked (see Supplementary Material for further details). Optimism was estimated for model coefficients, derived odds ratios and C-statistics. During the same procedure, it was also evaluated how much lower sensitivity and higher prevalence would be due to optimism, when applying a cutoff probability above which patients are classified as at risk. For this evaluation cutoff probabilities were selected for which either sensitivity or prevalence corresponded to the basic two-predictor model.
Results
Probabilities of 3GC-R EB bacteraemia were 0.4% (n = 90) for the community-onset infection cohort (22,506 episodes) and 1.0% (n = 82) for the hospital-onset infection cohort (8,110 episodes). These case populations were matched to 360 community-onset control episodes and 328 hospital-onset control episodes (Table 2). Multiple selection of individual patients, albeit with different episodes, as case and/or control were allowed and occurred 8 times within the community-onset, and 9 times within the hospital-onset dataset. Isolated pathogens from blood cultures and initial antibiotic therapy are presented in Supplementary Tables 4 and 5.
Community-onset infection
The final simplified model for predicting 3GC-R EB bacteraemia in community-onset infection (Table 3) showed adequate discrimination (C-statistic = 0.808 (95% CI 0.756-0.855), optimism-corrected 0.775 (95% CI 0.705-0.839)) and calibration (Supplementary Figure 1). In sensitivity analyses, a model variant that included both prior use of cephalosporins or fluoroquinolones, and prior use of a residual category of antibiotics (instead of any antibiotic use in the final simplified model) was deemed most parsimonious (Supplementary Tables 6 and 7).
The derived scoring system (Table 4) had a performance similar to the original model (Figure 1a; C‐ statistic 0.807 (95% CI 0.756-0.855), not corrected for optimism). Figure 2a depicts the trade-off between sensitivity and prevalence at different cutoffs for being at risk of 3GC-R EB bacteraemia, compared to using the prior identification model (sensitivity 24.4% and prevalence 2.8%), and the two-predictor model (sensitivity 53.9% and prevalence 21.5%).
For instance, patients with a score of 120 or higher would have a probability of 1.7% (positive predictive value) of having 3GC-R EB bacteraemia, but when using 120 as a cutoff 45.7% of all patients with 3GC-R EB bacteraemia would be missed (1 – sensitivity). This sensitivity (or proportion missed) is comparable to the simpler two-predictor model; however, the scoring system reduces eligibility for carbapenem use (prevalence) by approximately 40%, from 21.5% to 12.8%.
Bootstrapping of the underlying model indicated that when applying this cutoff in a future patient population some performance loss should be expected due to model optimism. The optimism-corrected sensitivity for future populations was 6.2 percentage points lower, whereas a change in prevalence was hardly noticeably (Table 6). Similarly, when basing the probability cutoff on the prevalence of the two-predictor model within the study sample, prevalence is expected to be robust in future populations, but a decrease in sensitivity of approximately 4.0 percentage points should be anticipated.
Hospital-onset infection
Discrimination and calibration also appeared adequate for the final simplified model predicting 3GC-R EB bacteraemia in hospital-onset infection (Table 5; C-statistic = 0.842 (95% CI 0.793-0.886), optimism-corrected 0.811 (95% CI 0.742-0.873); Supplementary Figure 2). Sensitivity analyses revealed better performance when including more variables, especially those related to antibiotic use and the suspected source of infection, and a model including 13 instead of 9 parameters had the lowest AIC (Supplementary Tables 10 and 11).
The derived simplified scoring system (Table 4) performed very similar to the original model (Figure 1b; C-statistic 0.842 (95% CI 0.794-0.887), not corrected for optimism). In Figure 2b, sensitivity and prevalence at different scoring cutoffs are compared to using the prior identification model (sensitivity 35.4% and prevalence 5.2%), and the two-predictor model (sensitivity 79.3% and prevalence 52.8%).
Patients with scores of 110 or higher would have a positive predictive value for 3GC-R EB bacteraemia of 3.1%, and with this cutoff 18.5% of all patients with 3GC-R EB bacteraemias would be missed, similarly to the two-predictor model. Yet, carbapenem eligibility would be reduced with almost 50% (27.0% vs. 52.8%).
In this scenario, bootstrapping indicated that sensitivity in future patient populations should again be expected to be somewhat lower (-5.3%; Table 6). If a cutoff were however based on prevalence of the two-predictor model, substantially compromised performance in future patients is not likely to occur.
Discussion
We developed prediction models to more accurately identify patients with bacteraemia caused by 3GC-R EB bacteraemia among those in whom empiric intravenous antibiotic therapy aimed at Gram-negatives is initiated. The use of the derived scoring systems could improve appropriateness of empiric antibiotic therapy and reduce unnecessary use of broad-spectrum therapy. Compared to a basic model incorporating only prior 3GC-R EB identification and exposure to cephalosporins and/or fluoroquinolones, eligibility for empiric carbapenem use could be reduced by approximately 40% while maintaining a similar risk of missing patients with 3GC-R EB bacteraemia.
With a global emergence of antibiotic resistance, physicians must assess the risks of missing resistant causative pathogens when starting empiric antibiotic treatment [20]. Risk avoidance, albeit imaginable in many situations, is one of the driving forces for broad-spectrum antibiotic use, fuelling the global pandemic of antimicrobial resistance. Better prediction rules for infections caused by antibiotic-resistant pathogens are therefore needed. Prediction scores have been developed for Gram-negative bacteraemia in septic patients [21], carriage of or infection with ESBL-producing Enterobacteriaceae at hospital admission [1,22,23], and distinguishing bacteraemia with ESBL-or carbapenemase-producing pathogens from bacteraemia with susceptible Enterobacteriaceae [5–7]. Yet, guidance on incorporating the risk of 3GC-R EB in selecting empiric antibiotics is currently lacking. For clarity, 3GC-R EB bacteraemia is a subset of Enterobacteriaceae bacteraemia, which is a subset of all bacteraemia episodes. Risk factors for any of these overarching categories may alter the probability of bacteraemia caused by antibiotic-resistant Enterobacteriaceae. This emphasizes the need of selecting a clinically meaningful patient population when deriving a prediction rule. We therefore focused on patients receiving initial antibiotic therapy aimed at Enterobacteriaceae, rather than selecting patients that had, in retrospect, bacteraemia. The differences in predictors for community-onset and hospital-onset bacteraemia underscore the relevance of distinguishing both entities in clinical prediction of infections caused by resistant Enterobacteriaceae.
As expected, prior identification of 3GC-R EB was the strongest predictor in both models. Identification was mostly based on previous clinical cultures, rather than on screening for carriage. The latter is in the Netherlands only practiced in intensive care units and for highly selected risk groups. Naturally, more screening will further increase the sensitivity of this predictor for bacteraemia with 3GC-R EB. Yet, as infection rates among colonized patients are low [24,25], it is unsure whether positive predictive values of models will improve. In fact, if low-risk carriers would be identified by screening more frequently than high-risk carriers, positive predictive values might even decline.
We applied sophisticated modelling techniques, including multiple imputation and internal validation by means of bootstrapping. The latter resulted in optimism-adjusted odds ratios and C-statistics, giving insight in values expected when applying models to an external cohort. Expected performance loss when selecting specific probability cutoffs for clinical use has also been calculated (Table 4). Nevertheless, we still recommend prospective external validation of the models before clinical implementation, for several reasons. First, even after shrinkage, optimism may still be present, as some steps could not be replicated in the bootstrap procedure, such as aggregation after observing similar associations with the outcome, simplification of continuous variables to linear predictors, and derivation of a scoring system. Second, explorative model variants indicated that predictive performance could be enhanced, albeit at the cost of including more determinants. The optimism of these variants has not been calculated, and may well offset potential increases in performance observed in this study sample in external validation. Second, the current study relied on data available in medical charts. We used pragmatic in‐ and exclusion criteria, which might not fully reflect intended clinical use, and as data collection was not blinded for outcome, information bias is not excluded. Moreover, potentially relevant predictors, especially for community-onset infection, such as international travel, animal contact, known colonization in household members, and dietary preferences could not be collected. The same holds for determination of colonization pressure, which might be a relevant predictor for hospital-onset infections. Third, although prevalence of ESBL in Enterobacteriaceae infection in the Netherlands, especially in the community setting, does not differ considerably from other Western European countries [26–29], it is worthwhile to evaluate performance of risk factors in settings with varying ESBL prevalence.
A study limitation is that the outcome was restricted to bacteraemic episodes, not including non-bacteraemic infections caused by 3GC-R EB, and physicians should be aware of this when applying the score in clinical practice. Non-bacteraemic 3GC-R EB infection are more common than bacteraemic infections in patients being empirically treated [9], but with an overall prevalence of <5%, they will not have had a substantial impact on the composition of control groups. Future studies may consider classifying these infections as outcomes. However, due to the more benign course, initial treatment with carbapenems may not have a high priority in non-bacteraemic infections. Ultimately, using such broad-spectrum antibiotics as initial therapy is futile if patients would not actually benefit from them.
Another limitation of our study is that empiric coverage of 3GC-R EB is just one aspects of selection of appropriate empiric therapy. Other potential pathogens and resistance mechanisms, such as Pseudomonas aeruginosa, might justify alterations in empiric treatment even in the absence of risk factors for 3GC-R EB. In some countries, high incidences of infections with carbapenemase-producing Enterobacteriaceae may limit usefulness of our models. On the other hand, escape therapy for 3GC-R EB might not necessarily involve carbapenems, due to underlying resistance mechanisms other than ESBL, or favourable patterns of co-resistance. Ideally, frameworks for selecting empiric therapy should evaluate the probability of success of many different antibiotic agents. An example of such an approach is TREAT, an automated system for recommending antibiotic treatment based on, amongst others, patient and infection characteristics and local epidemiology [30]. TREAT can predict the presence of Gram-negative causative pathogens in infection with some accuracy [31], but performance with regard to resistant variants remains unknown. However, TREAT has not been widely adopted, and simple prediction rules may be easier to incorporate into clinical practice.
In conclusion, identification of only those patients with an infection caused by 3GC-R EB amongst all patients that need empiric antibiotic therapy remains a daunting task. This is reflected by the fact that an acceptable level of empiric carbapenem use is likely to be associated with a considerable fraction of 3GC-R EB bacteraemias being missed. Yet, the prediction rules developed here are the first to truly quantify this trade-off for the appropriate patient domain. In addition, they offer considerable improvement in detecting such patients as compared to guidelines currently in place, and as such, they provide useful starting points for optimizing empiric antibiotic strategies.
Transparency declaration
W.C.R. has received research funding from Merck Sharp & Dohme (MSD). All other authors declare no conflicts of interest. This work was supported by a research grant from the Netherlands Organisation for Health Research and Development (project number 205200007) to H. S. M. A.
We would like to thank Evelien Stevenson and Tim Deelen for assistance with the data collection, and Maaike van Mourik and Loes de Bruin for assistance with the statistical analysis.
Results from this study were presented at the 26th European Congress of Clinical Microbiology and Infectious Diseases, Amsterdam, the Netherlands (9-12 April 2016, E-poster eP0083).