Sepsis is a leading global cause of preventable morbidity and mortality.^1–4 Early diagnosis, administration of antimicrobials and source control are the cornerstones of management and have been shown to increase survival.^5–8 Sepsis is defined as life-threatening organ dysfunction caused by a dysregulated host response to acute infection⁹ and is associated with a significant clinical and rising financial burden.¹⁰ Early diagnosis of sepsis in the hospital population is hampered by the absence of a gold standard diagnostic test and the need to collate diverse clinical and laboratory features to prompt clinical suspicion and distinguish sepsis from non-infectious diseases.^11,12

The introduction of electronic medical record (eMR) systems has provided a unique opportunity for automated early warning alerts for the potential presence of sepsis.^13–21 Numerous rule-based electronic sepsis alert systems (ESAS), often utilising outdated sepsis definitions, have been trialled in emergency departments and inpatient settings.²² While these studies have primarily focused on outcome measures such as mortality, hospital length of stay (LOS), and time to antimicrobial administration, few have assessed the diagnostic accuracy of an ESAS as a reliable sepsis surveillance tool.^14,15,18,22

Moreover, many of these alerts rely on criteria based on the systemic inflammatory response syndrome (SIRS) rather than the Sepsis-3 international definition of sepsis, creating a highly sensitive tool but one with a low positive predictive value (PPV).²³ This has been shown to lead to the misdiagnosis of sepsis, inappropriate administration of antimicrobials¹⁷ and potentially contributes to alert fatigue amongst clinicians.^24,25 Despite efforts to optimise alert criteria, balancing sensitivity and specificity remains a challenge.¹⁹

Recent advances in machine learning (ML) and artificial intelligence (AI) offer promising alternatives to rule-based systems. However, before ML-based approaches can be widely adopted, it is essential to rigorously evaluate the performance of existing rule-based systems in real-world clinical settings to establish baseline performance and identify specific areas for improvement.²⁶

We sought to evaluate the diagnostic accuracy of a rule-based ESAS that was designed in accordance with the latest international sepsis definition. Our study was conducted in a metropolitan teaching hospital in Australia equipped with a comprehensive hospital-wide eMR system.

Methods

Electronic Sepsis Alert System

The Millennium™ (Cerner Corp., Kansas City, MO) eMR was introduced throughout the hospital in October 2017 and included a third-party electronic sepsis surveillance and alerting system.²⁷ The rule-based sepsis alert system was redesigned to align with the Sepsis-3 international definition of sepsis.⁹ It continuously monitors patient vital signs and laboratory data collected in real-time by the eMR and compares these data with predetermined diagnostic criteria consistent with the presence of sepsis.

Two levels of alerts - Possible Sepsis Alert (PSA) and Sepsis Alert (SA) - are triggered according to the number and severity of clinical criteria present within a specific time window. At least three minor criteria must reach threshold values to trigger a PSA; whereas an SA is triggered by the presence of any two minor criteria plus at least one major feature suggesting organ dysfunction (Table 1). The presence of either alert then triggers an automated message to the bedside nurse, as well as to the treating doctor in the event of a SA. This message describes the clinical criteria present with the date/time stamp. In addition, the eMR prompts the primary nurse with the option and guidance to initiate investigations and treatment for suspected sepsis while waiting for assessment by the medical team. The eMR also provides the medical team with an interactive pathway (PowerPlan) for investigation and treatment of suspected sepsis. These data were collected prospectively and stored in the eMR.

Study setting, Population and Design

This single-centre observational study was undertaken at a university-affiliated hospital in Australia with 25,000 multiday-stay admissions per annum, a 24-hour cardiac catheter laboratory, 10-bed Intensive Care Unit and comprehensive acute inpatient specialty services. Inclusion criteria were all adult (≥ 18 years) multiday-stay inpatient admissions between January 1st, 2018, and December 31st, 2019, inclusive. Paediatric admissions (< 18 years) and Emergency Department (ED) attendances that were not admitted to an inpatient unit were excluded from the study. Sepsis alerts triggered during ED assessment and resuscitation were excluded from our analysis, as physiological derangements from various infectious and non-infectious conditions may mimic sepsis and confound the results.

All available data for PSA and SA including clinical triggers, timing, interventions, outcomes and date of hospital admission were extracted from the eMR and linked with administrative diagnosis coded data containing International Classification of Diseases and Health Related Problems 10th edition – Australian Modification (ICD-10-AM) using unique patient identifiers and admission and discharge dates. The presence of either sepsis alert (PSA or SA) at any time during the inpatient acute care phase were established. Where multiple alerts for the same patient occurred during a single hospital episode, only the first PSA and first SA were included.

The presence or absence of sepsis was determined from the clinical diagnoses coded in the administrative dataset. This is based on a published (“synchronous”) methodology that uses ICD-10-AM diagnosis codes and was validated against the reference standard of clinical chart review.²⁸ Sepsis was deemed to be present if at least one diagnosis code for an acute infection was accompanied by a second, synchronous diagnosis consistent with acute organ dysfunction. This method has been shown to yield excellent positive and negative predictive values and outperforms previously published (“explicit” and “implicit”) methods for identifying sepsis in administrative data. Note that the explicit term “sepsis,” found in many ICD-10-AM diagnoses, is alone insufficient and poorly predictive of sepsis.²⁹

Clinical (acute and comorbid) diagnoses and demographic characteristics (age, sex, ethnicity, admission source) were extracted from the administrative dataset. The burden of comorbid disease was quantified using both the Charlson and Elixhauser methods³⁰ and the presence of clinical frailty using the Gilbert method.³¹

Patient and episode identifiers were retained for data linkage of eMR and administrative datasets and then removed prior to analysis. During the study period, clinical staff were informed of all sepsis alerts (as described above) and all management decisions, including response to any sepsis alert, were determined by the treating team with usual care.

This study was conducted in accordance with the National Statement on Ethical Conduct in Human Research (NHMRC, 2007, updated 2018) and the principles of the Declaration of Helsinki (2013). The Eastern Health Human Research Ethics Committee approved this investigation (LR68-2018) and the need for patient consent was waived due to the unblinded and observational methodology, analysis of de-identified data and reporting of aggregated results.

From January 2018 to December 2019 there were 149,075 multiday-stay admissions (Fig. 1). Following the exclusion of paediatric records, 142,053 (95%) inpatient admissions were included in the analysis. There were 25,619 (18.0; 95% CI 17.8–18.3%) admissions associated with a diagnosis of acute infection at any time during the hospital episode and 5,058 (3.6%) patient admissions were classified as clinical sepsis, giving a prevalence rate of 3.56 (95% CI, 3.46–3.66) per 100 admissions.

The median lead time to the first sepsis alert was 2 days [IQR, 1–6] (Appendix; supplementary table 1). A comparison of the study cohorts which did and did not trigger any sepsis alert (PSA and SA) are provided in Table 2 (and Appendix; supplementary table 1). Notably, those who experienced any sepsis alert were older (median, 69 [IQR, 53–79] v 64 [IQR, 45–78] years; P < 0.001), more likely to follow an emergency admission (87% vs 43%; P < 0.001) and experienced a longer LOS (mean, 7.4 [SD, 12.2] v 1.8 [SD, 4.3]; P < 0.001) compared to those who did not trigger any sepsis alert. Both Elixhauser and Charlson comorbidity scores were significantly higher in those that triggered any sepsis alert (73% v 48%; P < 0.001 and 48% v 24%; P < 0.001, respectively). Furthermore, those who triggered a sepsis alert were more frequently admitted to ICU (13% vs 2%; P < 0.001) or died in-hospital (6% vs 0.8%; P < 0.001) than those with no sepsis alert.

Performance metrics for the sepsis alert system are depicted in Fig. 2 and summarised in Tables 3 and 4. The sensitivity and PPV for the presence of sepsis were 26.3% [95% CI, 25.1–27.6%] and 33.2% [95% CI, 31.7–34.7%] while the specificity and NPV were 98% [95% CI, 98-98.1%] and 97.3% [95% CI, 97.2–97.4%], respectively. The AUROC of 0.62 suggests an inefficient test.

Sensitivity analyses in subgroups with acute infection, or those aged > 65years, did not reveal any improvement in performance of sepsis alerts (Appendix; supplementary table 2). Some improvement in sensitivity 91.9% [95% CI, 89.4–94%] and PPV 44.5% [95% CI, 41.7–47.4%] was identified in the subgroup following inter-hospital transfer.

The green circle denotes admissions classified as clinical sepsis, the blue circle denotes Sepsis Alerts (SA), and the red circle denotes Possible Sepsis Alerts (PSA). The grey square represents the total inpatient study population. Overlap between circles indicates true-positive alerting, while sepsis cases outside the alert circles represent missed cases (false negatives) and alert activations outside the sepsis circle represent false positives. Circle size is proportional to the number of admissions in each group.

We screened nearly 150,000 consecutive hospital separations and their corresponding admission data identifying 18.0% with an acute infection and 3.6% with sepsis. Analysis of over 4,000 automated sepsis alerts generated by the hospital eMR revealed that the alert algorithm had a poor PPV (33.2% [95% CI, 31.7–34.7%]) despite a high NPV (97.3% [95% CI, 97.2–97.4%]) for the presence of sepsis. The automated sepsis alerts did not perform with sufficient reliability as a clinical prompt for sepsis.

In response to the results of this study and considering the impending COVID-19 pandemic, the study hospital elected to discontinue the automated sepsis alert system. Retaining a low sensitivity alert system may have led to missed sepsis cases and a delay in key interventions. This case illustrates that technical deployment of health information technology must be accompanied by ongoing performance monitoring and a readiness to modify or withdraw systems when warranted. These observations are relevant to the future development of automated clinical alerts, including next-generation systems that may incorporate adaptive or machine-learning-based approaches.

Despite its poor performance in identifying sepsis, the algorithm did appear to identify a patient cohort with a higher risk of prolonged hospital stay, transfer to ICU or in-hospital death (Table 2). Therefore, similar automated alert systems may help clinicians identify a cohort at higher risk of clinical deterioration and adverse events in line with Standard-8 of the Australian National Healthcare Standards.³³ An expanded version of the existing algorithm may find clinical application for this purpose and warrants further investigation.

Sepsis diagnosis requires the triangulation of vital signs and laboratory parameters together integrated with clinical findings that indicate the presence of infection, while also excluding confounding factors such as non-infectious inflammatory states that can mimic severe infection.¹¹ The low sensitivity and PPV of this alert system are consistent with other electronic sepsis alerts developed for inpatient and critical care settings.^13,19–21 These findings reflect the non-specific nature of the diagnostic criteria for sepsis and its tendency to overlap with many non-infectious diseases that lead to significant organ dysfunction. The balance between early warning and excessive over-treatment is challenging.¹⁷

While our study demonstrated that this ESAS had high specificity, its low sensitivity and PPV limit its clinical reliability. This reflects inherent limitations of rule-based systems that depend on fixed thresholds and predefined criteria. These systems lack contextual awareness, perform poorly in heterogeneous clinical settings, and are unable to adapt to evolving patient data or changing sepsis definitions.^13,34 Our findings align with a growing body of evidence demonstrating that rule-based approaches are insufficient for complex clinical predictions like sepsis, where subtle patterns and temporal relationships are critical.¹³

In contrast, machine learning (ML)-based approaches have shown promise in overcoming many of these challenges. ML models can incorporate an array of continuous and categorical variables, model non-linear relationships, and detect subtle temporal patterns that may precede overt clinical deterioration. Several studies have demonstrated that ML algorithms can outperform rule-based systems, achieving higher sensitivity and PPV, with improved AUROC values when predicting sepsis.^26,35 When combined with timely provider review, ML-based early warning systems have shown potential to reduce mortality and hospital length of stay in sepsis patients.^34,36 Additionally, these systems may offer significant cost-saving potential by enabling earlier intervention and reducing the need for ICU admissions.³⁷

This study provides a real-world evaluation of a widely deployed sepsis alert system and demonstrates the essential role of post-implementation assessment in health informatics. Our findings serve as an important benchmark for evaluating rule-based approaches and provide a foundation for developing more sophisticated, adaptive systems. However, further research is warranted to determine whether ML-based alerts can substantially improve diagnostic performance while avoiding new risks such as alert fatigue, lack of clinical transparency, or inappropriate treatment of non-infectious conditions under the mistaken presumption of sepsis.^17,24

The strengths of this study include a large study population and whole-of-hospital cohort with a sufficiently high prevalence of sepsis (3.6%). The two-month eMR run-in period, prior to study commencement, permitted time for education of staff including familiarity with the sepsis alert system. Study data were collected over a two-year period capturing seasonal and case mix variation in the frequency of sepsis. Data collection and provisional analysis were completed prior to the SARS-COV19 pandemic. To date, this is the largest published study to address the clinical validity of automated sepsis alerts in Australia.

There are several important limitations in our study. The findings of this single centre study may not necessarily be generalisable to other sites. Clinical diagnosis and treatment following a sepsis alert were left to the judgement of the treating team and the study hospital does not employ protocolised early-goal directed therapy for sepsis. Additionally, the presence or absence of sepsis was determined from the clinical diagnoses coded in the administrative dataset. While the use of ICD-10-AM coding data has been shown to over or under report cases of sepsis when compared to the gold standard of prospective chart review for objective markers of infection,^3,29 the method utilised in our study outperforms other methods for identifying sepsis in administrative data.²⁸. Although this inaccuracy may affect the calculated performance metrics of the electronic sepsis alerts potentially leading to deviations from the true diagnostic accuracy that would be achieved through comprehensive chart review, our methodology permitted a larger study population over a longer duration with limited research resources and the (SARS-COV19) pandemic imminent.

We investigated the performance and reliability of an automated clinical alert system for early identification of in-hospital sepsis. While the algorithm appears to be highly specific, it lacked sufficient sensitivity for clinical purposes. Further research and clinical validation are required to refine these potentially attractive systems to improve patient care without increasing unnecessary workload burden or exposing patients to risks of unnecessary investigations and treatment.