If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Emergency Department, Research Center for Emergency Medicine, Aarhus University Hospital, Aarhus, DenmarkDepartment of Clinical Medicine, Aarhus University, Aarhus, Denmark
Reprint Address: Michael L. Dalsgaard, MSc, Emergency Department, Research Center for Emergency Medicine, Aarhus University Hospital, Palle Juul-Jensens Boulevard 99, J103, DK-8200 Aarhus N, Denmark.
Footnotes
1 These authors contributed equally.
Affiliations
Emergency Department, Research Center for Emergency Medicine, Aarhus University Hospital, Aarhus, DenmarkDepartment of Clinical Medicine, Aarhus University, Aarhus, Denmark
Emergency Department, Research Center for Emergency Medicine, Aarhus University Hospital, Aarhus, DenmarkEmergency Department, Regional Hospital Horsens, Horsens, Denmark
Physicians tend to overestimate patients’ pretest probability of having bacteremia. The low yield of blood cultures and contaminants is associated with significant financial cost, as well as increased length of stay and unnecessary antibiotic treatment.
Objective
This study examined the abilities of the National Early Warning Score (NEWS), the Quick Sequential Organ Failure Assessment (qSOFA), the Modified Sequential Organ Failure Assessment (mSOFA), and two versions of the causal probabilistic network, SepsisFinder™ (SF) to predict bacteremia in adult emergency department (ED) patients.
Methods
This cohort study included adult ED patients from a large urban, academic tertiary hospital, with blood cultures obtained within 24 h of admission between 2016 and 2017. The outcome measure was true bacteremia. NEWS, qSOFA, mSOFA, and the two versions of SF score were calculated for all patients based on the first available full set of vital signs within 2 h and laboratory values within 6 h after drawing the blood cultures. Area under the receiver operating characteristic curve (AUROC) was calculated for each scoring system.
Results
The study included 3106 ED patients, of which 199 (6.4%) patients had true bacteremia. The AUROCs for prediction of bacteremia were: NEWS = 0.65, qSOFA = 0.60, SF I = 0.65, mSOFA = 0.71, and SF II = 0.80.
Conclusions
Scoring systems using only vital signs, NEWS, and SF I showed moderate abilities in predicting bacteremia, whereas qSOFA performed poorly. Scoring systems using both vital signs and laboratory values, mSOFA and especially SF II, showed good abilities in predicting bacteremia.
). Bacteremia, confirmed by a positive blood culture, is common, with an incidence rate of 114–166 per 100,000 person-years, among patients treated in public hospitals in the North Denmark Region (
Temporal changes in the incidence and 30-day mortality associated with bacteremia in hospitalized patients from 1992 through 2006: a population-based cohort study.
). The tendency to overestimate pretest probability may be due to a sparse set of guidelines clearly describing which patients should have blood cultures drawn (
). The low yield of blood cultures and contaminants is associated with significant financial costs, as well as increased length of stay and unnecessary antibiotic treatment (
Prediction models for risk stratification of patients with suspected bacteremia have been developed and tested to support physicians in deciding whether to obtain blood cultures, but none are widely used in clinical practice (
). This raises the question of whether already implemented scoring systems used for other purposes with readily available values could also predict bacteremia, such as the National Early Warning Score (NEWS), the Quick Sequential Organ Failure Assessment (qSOFA), and the Modified Sequential Organ Failure Assessment (mSOFA) (
). In addition, a causal probabilistic network (CPN), SepsisFinder™ (SF), which is not yet in routine clinical use, has been suggested for predicting bacteremia (
). This study's aim was to examine the abilities of the scoring systems—NEWS, qSOFA, mSOFA, and SF—to predict bacteremia in adult patients admitted to the emergency department (ED) who had blood cultures drawn.
Materials and Methods
Study Design and Setting
This was a retrospective cohort study of adult patients admitted to a large urban, academic tertiary ED at Aarhus University Hospital (AUH), Aarhus, Denmark, who had a blood culture drawn. The ED is a short-stay unit (usually up to 24 h), has approximately 63,000 annual patient visits, and provides 24-h emergency care to all acute patients, except those with myocardial infarction or stroke and women in labor. The health care system in Denmark is free and tax-supported for patients with a Danish Civil Personal Registration (CPR) number.
The Danish Patient Safety Authority (No. 31-1522-47) and the Danish Data Protection Agency (No. 1-16-02-371-18) approved the study. According to Danish law, ethics committee approval was not required.
Study Population
We included adult ED patients (age ≥ 18 years) with a valid Danish CPR number who had a blood culture performed in the ED between July 1, 2016 and June 30, 2017, within 24 h of ED arrival. Patients discharged or transferred to other departments within the 24-h period but who had a blood culture performed in the ED were kept in the cohort. We considered a performed blood culture as a proxy for suspicion of bacteremia. The attending physician decided whether to draw a blood culture in the ED without knowledge of this study. Only the first admission with blood cultures drawn during the inclusion period was included. The patient demographics, the first full set of vital signs within 2 h, and the laboratory values within 6 h of the blood culture being drawn were retrieved from the electronic health record and from the microbiology data system for all adult patients admitted to the ED.
Scoring Systems
We used the locally implemented version of NEWS called “Tidlig Opsporing af Kritisk Sygdom (TOKS),” which is used in EDs and on regular wards to monitor adult patients after triage in Central Denmark Region. It is based on respiration rate, saturation, systolic blood pressure, pulse, temperature, and level of consciousness using the alert, voice, pain, and unresponsive scale (AVPU). Each vital sign is weighted and summed to a final score (Appendix A). To calculate a TOKS score as soon as possible, values were taken during triage, converting the Glasgow Coma Scale (GCS) to AVPU (GCS 15 = A, GCS 14 = V, GCS 9–13 = P, and GCS ≤ 8 = U) (
). The TOKS score is subsequently referred to as NEWS in this study.
The qSOFA uses simple bedside criteria to identify adult patients with suspected infection who are likely to have poor outcomes, and includes systolic blood pressure ≤ 100 mm Hg, a respiratory rate ≤ 22 breaths/min, and a GCS score < 15. A positive score is defined as two or more out of three criteria (Appendix B) (
The Danish Society of Infectious Diseases has developed a modified version of the Sequential Organ Failure Assessment (SOFA), called mSOFA, which is suitable for patients outside of the intensive care unit. The mSOFA is based on the GCS, PaO2 (kPa), systolic blood pressure (mm Hg), bilirubin (μmol/L), creatinine (μmol/L), and platelet count (× 103/μL). A score from 0 to 4 based on a set of reference intervals is given for each organ with a final sum score (Appendix C) (
Rekommandationer for initial behandling af sepsis og septisk shock.
Dansk Selskab for Infektionsmedicin;.2021; ([Danish]): 11
). Ventilator and vasopressor treatments are not included in the calculations.
All variables had to be available prior to calculation in the above-mentioned scoring systems.
The SF is a stochastic model that uses CPN as a framework for representing medical knowledge that can be coded as dependence between stochastic variables (
). To predict the probabilities, SF uses vital signs and laboratory values, but does not depend on a specific number of variables being present to predict the probability of bacteremia (
). Therefore, two versions of SF were investigated: SF I, which only used vital signs, and SF II, which used vital signs and laboratory values, if and when available (Appendix D).
Bacteremia
Each patient had one or more sets of blood cultures drawn, consisting of one aerobic and one anaerobic bottle, with a nominal blood sample volume of 10 mL per bottle (BacT/Alert; bioMérieux, Marcy l'Etoile, France). A set was considered positive when bacteria grew in at least one bottle.
Outcome
The outcome was true bacteremia, defined as bacterial or fungal growth in blood cultures deemed to have an etiological role on the basis of joint clinical and microbiological assessment. All blood culture isolates were classified prospectively as either bacteremia, possible contamination, or definite contamination. In accordance with Weinstein et al., coagulase-negative Staphylococci, Corynebacterium spp., Bacillus spp., and Propionibacterium acnes were considered definite contaminants unless isolated from two or more separate blood culture sets (
The clinical significance of positive blood cultures: a comprehensive analysis of 500 episodes of bacteremia and fungemia in adults. I. Laboratory and epidemiologic observations.
). They were considered true bacteremia or possible contamination if two of four bottles were positive for the above-mentioned bacteria. A microbiologist (MTR) reviewed all true bacteremia and possible contaminations. If a blood culture was determined as exclusively positive for contaminants, the blood culture was considered negative. If a blood culture consisted of both a contaminant and a significant bacterium, it was considered positive. We included all patients who had a blood culture drawn in the ED irrespective of the final result of the blood culture.
Statistical Analysis
We used means with standard deviations, medians with interquartile ranges, and proportions as appropriate for descriptive statistics. We examined the scoring systems’ abilities to predict bacteremia using sensitivity, specificity, and positive and negative predictive values (PPV and NPV) with 95% confidence intervals (95% CI). We used Youden's index to determine which cut-off values to report in this study and calculated the area under the receiver operating characteristic curve (AUROC) using the sum score given by the scoring systems (
). We report the average scores for bacteremia and nonbacteremia patients. We analyzed data using STATA 16.1 (StataCorp, College Station, Texas).
Results
Population Characteristics
During the 1-year study period, 4200 patients had blood cultures drawn in the ED, and 3106 patients met the inclusion criteria. Among the 1094 excluded patients were 12 patients without a valid Danish CPR number, 252 patients who were not admitted, 151 patients whose blood culture was drawn later than 24 h after admission, and 679 readmissions during the inclusion period. No patients were excluded due to missing data points.
Patient characteristics are reported in Table 1, stratified by bacteremia and nonbacteremia.
Table 1Patient Characteristics
Bacteremia (n = 199)
Nonbacteremia (n = 2907)
Demographics, mean (SD)
Age, years
69.9 (15)
65.2 (20)
Male sex, n (%)
114 (57)
1,406 (48)
Vital signs, mean (SD)
Systolic blood pressure (mm Hg)
129 (26)
139 (26)
Mean arterial pressure (mm Hg)
91 (18)
98 (18)
Heart rate (beats/min)
99 (21)
92 (20)
Respiratory rate (breaths/min)
23 (7)
21 (7)
Temperature (°C)
38.4 (1)
37.8 (1)
Glasgow Coma Scale - score < 15, n (%)
42 (22)
406 (15)
SpO2 %, median [IQR]
96 [94–98]
97 [95–99]
Blood samples, median [IQR]
C-reactive protein (mg/L)
119 [40–213]
49 [12–114]
B-neutrophil leukocyte, fraction (%)
89 [84–92]
78 [70–85]
P-creatinine (μmol/L)
99 [77–152]
78 [63–102]
P-albumin (g/L), mean (SD)
30 (5)
33 (5)
P-bilirubin (μmol/L)
14 [8–21]
8 [6–13]
B-platelets (× 109/L)
190 [151–237]
241 [185–309]
Arterial blood gas, median [IQR]
P(aB)-pO2 (kPa)
9.9 [8.6–12.7]
10.1 [8.6–12.3]
P(aB)-lactate (mmol/L)
2.5 [1.7–4.3]
1.5 [1.1–2.4]
Admission & mortality, median [IQR]
Cause of admission: fever or infection, n (%)
65 (33)
584 (20)
Time in emergency department (hours)
21 [8–35]
21 [9–33]
Total admission time (hours)
150 [86–240]
73 [30–147]
Time until blood culture (min.)
42 [30–70]
49 [31–82]
Intensive care during admission, n (%)
30 (15)
117 (4)
Mortality during admission n (%)
13 (7)
112 (4)
30 days mortality from admission n (%)
24 (12)
257 (9)
SD = standard deviation; IQR = interquartile range.
A total of 199 patients (6.4%) had true bacteremia; 71 (36%) cases were due to Gram-positive bacteria, 104 (52%) were due to Gram-negative bacteria, and 24 (12%) patients had polymicrobial bacteremia. Seventy-three patients (2%) were considered contaminated, of which 70 (2%) were definitely contaminated and three (0.1%) had possible contaminated blood cultures, which were all regarded as negative blood cultures (Appendix E).
Performance of Scoring Systems
The patients with bacteremia had higher scores compared with patients with nonbacteremia (Table 2). The data availability of the variables for calculating the five scoring systems—NEWS, qSOFA, SF I, mSOFA, and SF II—are shown in Appendix F. The AUROC values for the prediction of bacteremia were as follows: NEWS = 0.65 (95% CI 0.60–0.70), qSOFA = 0.60 (95% CI 0.56–0.64), SF I = 0.65 (95% CI 0.60–0.69), mSOFA = 0.71 (95% CI 0.65–0.76), and SF II = 0.80 (95% CI 0.76–0.83) (Figure. 1).
Table 2Average Scores for Bacteremia and Nonbacteremia Patients
Bacteremia
Nonbacteremia
NEWS, median [IQR]
5 [2.5; 6]
3 [1; 4]
qSOFA, median [IQR]
1 [0; 1]
0 [0; 1]
SF I, mean (SD)
7.96% (4.7)
5.87% (2.7)
mSOFA, median [IQR]
3 [2; 5]
2 [1; 3]
SF II, mean (SD)
10.34% (5.2)
5.69% (3.6)
NEWS = National Early Warning Score; IQR = interquartile range; qSOFA = quick Sequential Organ Failure Assessment; SF = SepsisFinder; SD = standard deviation; mSOFA = modified Sequential Organ Failure Assessment.
Figure 1Area under the receiver operating characteristic (AUROC) curve for predicting bacteremia. *AUROC [95% confidence interval]. SF = SepsisFinder; mSOFA = modified Sequential Organ Failure Assessment; NEWS = National Early Warning Score; qSOFA = quick Sequential Organ Failure Assessment.
The calculated endpoints for all the cut-off values are shown in Appendix G. Table 3 presents the sensitivity, specificity, PPV, NPV, potential reduction in blood cultures, and missed bacteremia with the given cut-offs for NEWS, qSOFA, SF I, mSOFA, and SF II.
Table 3Performance of NEWS, qSOFA, SF I, mSOFA, and SF II
n (%) = number of patients with bacteremia under the cut-off value and the percentage of missed bacteremia based on the total number of bacteremia a given scoring system could calculate. NEWS = National Early Warning Score; qSOFA = quick Sequential Organ Failure Assessment; SF = SepsisFinder; mSOFA = modified Sequential Organ Failure Assessment; CI = confidence interval; PPV = positive predictive value; NPV = negative predictive value.
NEWS
5
576 (25.6)
52.1 [43.5-60.7]
76.1 [74.2-77.9]
12.7 [10.1-15.7]
96 [94.9-96.9]
1670 (74.4)
67/140 (47.9)
qSOFA
1
1367 (46.7)
62 [54.7-68.9]
54.4 [52.5-56.3]
8.7 [7.3-10.3]
95.3 [94.2-96.3]
1561 (53.3)
73/192 (38)
SF I
6.34%
931 (30)
52.8 [45.6-59.9]
71.6 [69.9-73.2]
11.3 [9.3-13.5]
95.7 [94.7-96.5]
2175 (70)
94/199 (47.2)
mSOFA
3
373 (37.7)
65.8 [54-76.3]
64.7 [61.5-67.8]
13.4 [10.1-17.3]
95.8 [93.9-97.2]
617 (62.3)
26/76 (34.2)
SF II
5.99%
1243 (40)
83.4 [77.5-88.3]
63 [61.2-64.8]
13.4 [11.5-15.4]
98.2 [97.5-98.8]
1863 (60)
33/199 (16.6)
*Youden's index was used to determine which cut-off values to report in this table.
† n (%) = number of patients ≥ the cut-off value and the percentage of the total number of times a given scorings system could be calculated.
‡ n (%) = number of patients under the cut-off value and the percentage of the total number of times a given scorings system could be calculated.
§ n (%) = number of patients with bacteremia under the cut-off value and the percentage of missed bacteremia based on the total number of bacteremia a given scoring system could calculate.NEWS = National Early Warning Score; qSOFA = quick Sequential Organ Failure Assessment; SF = SepsisFinder; mSOFA = modified Sequential Organ Failure Assessment; CI = confidence interval; PPV = positive predictive value; NPV = negative predictive value.
Based on the highest Youden's Index, the preferred cut-offs and endpoints for the scoring systems’ abilities in predicting bacteremia are reported in Table 3 and were as follows (95% CI); NEWS cut-off ≥ 5: sensitivity 52.1% (43.5–60.7%) and specificity 76.1% (74.2–77.9%); qSOFA cut-off ≥ 1: sensitivity 62% (54.7–68.9%) and specificity 54.4% (52.5–56.3%); SF I cut-off ≥ 6.34%: sensitivity 52.8% (45.6–59.9%) and specificity 71.6% (69.9–73.2%); mSOFA cut-off ≥ 3: sensitivity of 65.8% (54–76.3%) and specificity 64.7% (61.5–67.8%); and SF II cut-off ≥ 5.99%: sensitivity of 83.4% (77.5–88.3%) and specificity 63% (61.2–64.8%).
Discussion
This study's results suggest that scoring systems using both vital signs and laboratory values have better abilities in predicting bacteremia compared to scoring systems using only vital signs. Among the scoring systems using only vital signs, SF I and NEWS showed better abilities in predicting bacteremia compared to qSOFA. The scoring systems that also included laboratory values, both mSOFA and SF II, showed good abilities in predicting bacteremia. However, SF II was able to include more patients through its CPN by continuously predicting the likelihood of bacteremia using selected values as they became available. Of the five scoring systems, SF II showed the best abilities in predicting bacteremia with the highest sensitivity, NPV, and AUROC, along with the lowest percentages of missed bacteremia.
To predict bacteremia, scoring systems are used as tools supporting clinical suspicion, but should always be secondary to good clinical judgement. Previously developed prediction tools for the prediction of bacteremia in the ED do not seem to have been clinically implemented (
). Scoring systems like NEWS, qSOFA, and mSOFA have the advantage of being well-known bedside scoring systems in the clinical setting compared with a CPN such as SF I and SF II. Furthermore, mSOFA and SF II use laboratory values, which postpone the decision-making, raising the question of whether to postpone the blood draw for cultures until laboratory values are available. Laboratory values in EDs in the Central Denmark Region are available within a maximum of 2 h. Several possible consequences of postponing blood cultures include more blood sampling and laboratory analyses, changes in workflow and procedures, and delayed treatment, as national and international guidelines require blood cultures prior to starting antibiotic treatment (
). Nevertheless, this study's results showed that scoring systems using laboratory values had better abilities in predicting bacteremia compared with scoring systems that did not include laboratory values.
Usman et al. retrospectively investigated the original NEWS's applicability as an early predictor of severe sepsis and septic shock in an ED and found an AUROC of 0.91 (
). This supports the Danish version of NEWS's potential for predicting bacteremia, together with the AUROC of 0.65 for NEWS found in this study.
The qSOFA score had the lowest AUROC (0.60) among the five scoring systems in this study. This could be explained by the fact that qSOFA used the fewest variables and was the simplest scoring system of those compared in this study. In Otani et al.’s study, qSOFA score likewise showed a low AUROC of 0.59 when predicting bacteremia (
). The Surviving Sepsis Campaign recently published the International Guidelines for Management of Sepsis and Septic Shock 2021, recommending against using qSOFA compared with, among others, NEWS as a single screening tool for sepsis or sepsis shock (
). The mSOFA is highly comparable with and simpler to use in the ED setting than the regular SOFA score due to its use of systolic blood pressure instead of mean arterial pressure and PaO2 instead of PaO2/FiO2. The mSOFA achieved an AUROC of 0.71 (95% CI 0.65–0.76) based on the 990 patients in this study. Similar results have been shown in a recent study outside the intensive care unit, where the full SOFA score had an AUROC of 0.71 (95% CI 0.62–0.80) based on 427 patients (
). The comparability of the results may contribute to future comparisons of the two scoring systems and help emphasize mSOFA's ability to predict bacteremia.
The major advantage of SF is that it is not dependent on all variables being available for calculating a score. A new score can be calculated every time a new variable is added or changed during the admission process. However, SF's weakness, as shown in this study, is that to achieve a high level of discrimination for bacteremia patients, it must have laboratory values, which are often available only after requisition and sampling of blood culture. Different versions of SF are described in the literature, but to the best of our knowledge, SF has not been studied when limiting its use to include only vital signs, as in SF I.
In Andreassen et al.’s recent study on SF II, they found an AUROC of 0.75 (95% CI 0.70–0.76) in predicting bacteremia in a combined dataset from Denmark and Israel (
). The current study's version of SF II did not include chills, mental status, and age, but added PaO2 to compare with the later version of SF that Andreassen et al. used (
). The variation in AUROC values may also be influenced by different settings.
Machine learning, smart algorithms and networks, and other terms associated with artificial intelligence (AI) have become common in the health care literature (
). AI seems to have the potential to improve early identification of bacteremia in patients who may benefit from antibiotic administration. AI is not yet used in our ED setting; however, in the future, it seems very likely that it can be implemented for predicting and flagging patients at risk of specific events.
This study's results showed the potential of scoring systems to supplement clinical judgement, as guidelines are sparse and do not clearly describe which patients should have blood cultures drawn. A prospective study should be initiated to validate the scoring systems prior to use in clinical practice, because prospective studies validating prediction models often show lower predictive abilities than validation studies (
). For scoring systems with sparse evidence in predicting bacteremia, validation at other sites prior to implementation in clinical practice is required.
Limitations
This study has some limitations. There was a lack of clear guidelines for the decision to draw blood cultures. This may have resulted in patients with bacteremia not having blood cultures drawn and thus not being included in the study. In addition, the classification of bacteremia and nonbacteremia patients could be faulty with a misclassification bias, although this should be minimized in the study. The study's retrospective design did not permit evaluation of the potential harm caused by not drawing blood cultures from the patients with scores below the cut-offs. Furthermore, the admission, treatment, and consequences for the hypothetical missed bacteremia patients were not studied.
Conclusions
In conclusion, the five scoring systems—NEWS, qSOFA, SF I, mSOFA, and SF II—were able to predict bacteremia in adult patients who had blood cultures drawn in the ED. Using only vital signs, NEWS and SF I showed moderate abilities in predicting bacteremia, both with an AUROC of 0.65, whereas qSOFA was inferior (AUROC 0.60). Using both vital signs and laboratory values, mSOFA (AUROC 0.71), and especially SF II (AUROC 0.80), showed good abilities in predicting bacteremia. With an increasing focus on AI, SF II, especially, has potential.
Article Summary
1. Why is this topic important?
Physicians tend to overestimate patients’ pretest probability of having bacteremia, as guidelines are sparse. The low yield of blood cultures and contaminants is associated with significant financial costs, increased length of stay, and unnecessary antibiotic treatment.
2. What does this study attempt to show?
This study attempts to demonstrate National Early Warning Score (NEWS), quick Sequential Organ Failure Assessment (qSOFA), modified Sequential Organ Failure Assessment (mSOFA), and SepsisFinder's ability to predict bacteremia in adult emergency department patients.
3. What are the key findings?
The scoring systems that used vital signs and laboratory values showed better abilities in predicting bacteremia compared with scoring systems that only used vital signs.
4. How is patient care impacted?
Scoring systems can supplement physicians’ clinical judgment when deciding whether to perform a blood culture and start treatment.
Temporal changes in the incidence and 30-day mortality associated with bacteremia in hospitalized patients from 1992 through 2006: a population-based cohort study.
The clinical significance of positive blood cultures: a comprehensive analysis of 500 episodes of bacteremia and fungemia in adults. I. Laboratory and epidemiologic observations.
00531-5&id=gr1.jpg){kind=link}