Advertisement

Machine Learning Methods for Predicting Patient-Level Emergency Department Workload

      Abstract

      Background

      Work Relative Value Units (wRVUs) are a component of many compensation models, and a proxy for the effort required to care for a patient. Accurate prediction of wRVUs generated per patient at triage could facilitate real-time load balancing between physicians and provide many practical operational and clinical benefits.

      Objective

      We examined whether deep-learning approaches could predict the wRVUs generated by a patient's visit using data commonly available at triage.

      Methods

      Adult patients presenting to an urban, academic emergency department from July 1, 2016–March 1, 2020 were included. Deidentified triage information included structured data (age, sex, vital signs, Emergency Severity Index score, language, race, standardized chief complaint) and unstructured data (free-text chief complaint) with wRVUs as outcome. Five models were examined: average wRVUs per chief complaint, linear regression, neural network and gradient-boosted tree on structured data, and neural network on unstructured textual data. Models were evaluated using mean absolute error.

      Results

      We analyzed 204,064 visits between July 1, 2016 and March 1, 2020. The median wRVUs were 3.80 (interquartile range 2.56–4.21), with significant effects of age, gender, and race. Models demonstrated lower error as complexity increased. Predictions using averages from chief complaints alone demonstrated a mean error of 2.17 predicted wRVUs per visit (95% confidence interval [CI] 2.07–2.27), the linear regression model: 1.00 wRVUs (95% CI 0.97–1.04), gradient-boosted tree: 0.85 wRVUs (95% CI 0.84–0.86), neural network with structured data: 0.86 wRVUs (95% CI 0.85–0.87), and neural network with unstructured data: 0.78 wRVUs (95% CI 0.76–0.80).

      Conclusions

      Chief complaints are a poor predictor of the effort needed to evaluate a patient; however, deep-learning techniques show promise. These algorithms have the potential to provide many practical applications, including balancing workloads and compensation between emergency physicians, quantify crowding and mobilizing resources, and reducing bias in the triage process.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Emergency Medicine
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Traub SJ
        • Stewart CF
        • Didehban R
        • et al.
        Emergency department rotational patient assignment.
        Ann Emerg Med. 2016; 67: 206-215
        • Hodgson NR
        • Traub SJ.
        Patient assignment models in the emergency department.
        Emerg Med Clin North Am. 2020; 38: 607-615
        • Wiler JL
        • Poirier RF
        • Farley H
        • Zirkin W
        • Griffey RT.
        Emergency severity index triage system correlation with emergency department evaluation and management billing codes and total professional charges.
        Acad Emerg Med. 2011; 18: 1161-1166
        • Platts-Mills TF
        • Travers D
        • Biese K
        • et al.
        Accuracy of the emergency severity index triage instrument for identifying elder emergency department patients receiving an immediate life-saving intervention.
        Acad Emerg Med. 2010; 17: 238-243
        • Mistry B
        • Stewart De Ramirez S
        • Kelen G
        • et al.
        Accuracy and reliability of emergency department triage using the emergency severity index: an international multicenter assessment.
        Ann Emerg Med. 2018; 71 (e3): 581-587
        • Hinson JS
        • Martinez DA
        • Schmitz PSK
        • et al.
        Accuracy of emergency department triage using the Emergency Severity Index and independent predictors of under-triage and over-triage in Brazil: a retrospective cohort analysis.
        Int J Emerg Med. 2018; 11: 3
        • Ginsburg AD
        • e Silva LO
        • Mullan A
        • et al.
        Should age be incorporated into the adult triage algorithm in the emergency department?.
        Am J Emerg Med. 2021; 46: 508-514
        • Collins GS
        • Reitsma JB
        • Altman DG
        • Moons KG.
        Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): the TRIPOD statement.
        Br J Surg. 2015; 102: 148-158
        • Horng S
        • Greenbaum NR
        • Nathanson LA
        • McClay JC
        • Goss FR
        • Nielson JA.
        Consensus development of a modern ontology of emergency department presenting problems—the Hierarchical Presenting Problem Ontology (HaPPy).
        Appl Clin Inform. 2019; 10: 409-420
      1. Available at https://www.aclweb.org/anthology/P16-1002. Accessed April 10, 2022.
        • Wu S
        • Roberts K
        • Datta S
        • et al.
        Deep learning in clinical natural language processing: a methodical review.
        J Am Med Inform Assoc. 2020; 27: 457-470
        • Joseph JW
        • Leventhal EL
        • Grossestreuer AV
        • et al.
        Deep-learning approaches to identify critically Ill patients at emergency department triage using limited information.
        J Am Coll Emerg Physicians Open. 2020; 1: 773-781
      2. Available at https://www.kdd.org/kdd2016/papers/files/rfp0697-chenAemb.pdf. Accessed October 29, 2022.
      3. Kingma DP, Ba J. Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014 Dec 22. Available at: http://arxiv.org/abs/1412.6980. Accessed May 10, 2022.

        • Burke LG
        • Wild RC
        • Orav EJ
        • Hsia RY.
        Are trends in billing for high-intensity emergency care explained by changes in services provided in the emergency department? An observational study among US Medicare beneficiaries.
        BMJ Open. 2018; 8e019357
        • Vaccarino V
        • Rathore SS
        • Wenger NK
        • et al.
        Sex and racial differences in the management of acute myocardial infarction, 1994 through 2002.
        N Engl J Med. 2005; 353: 671-682
        • Steenblik J
        • Smith A
        • Bossart CS
        • et al.
        Gender disparities in cardiac catheterization rates among emergency department patients with chest pain.
        Crit Pathw Cardiol. 2021; 20: 67-70
        • Hvelplund A
        • Galatius S
        • Madsen M
        • et al.
        Women with acute coronary syndrome are less invasively examined and subsequently less treated than men.
        Eur Heart J. 2010; 31: 684-690
        • Bösner S
        • Haasenritter J
        • Hani MA
        • et al.
        Gender bias revisited: new insights on the differential management of chest pain.
        BMC Fam Pract. 2011; 12: 45
        • Di Carlo A
        • Lamassa M
        • Baldereschi M
        • et al.
        Sex differences in the clinical presentation, resource use, and 3-month outcome of acute stroke in Europe: data from a multicenter multinational hospital-based registry.
        Stroke. 2003; 34: 1114-1119
        • Natale JE
        • Joseph JG
        • Rogers AJ
        • et al.
        Cranial computed tomography use among children with minor blunt head trauma: association with race/ethnicity.
        Arch Pediatr Adolesc Med. 2012; 166: 732-737
        • Johnson TJ
        • Weaver MD
        • Borrero S
        • et al.
        Association of race and ethnicity with management of abdominal pain in the emergency department.
        Pediatrics. 2013; 132: e851-e858
        • Gardner CS
        • Jaffe TA
        • Nelson RC.
        Impact of CT in elderly patients presenting to the emergency department with acute abdominal pain.
        Abdom Imaging. 2015; 40: 2877-2882
        • Leuthauser A
        • McVane B.
        Abdominal pain in the geriatric patient.
        Emerg Med Clin North Am. 2016; 34: 363-375
        • Goske MJ
        • Applegate KE
        • Boylan J
        • et al.
        The ‘Image Gently’ campaign: increasing CT radiation dose awareness through a national education and awareness program.
        Pediatr Radiol. 2008; 38: 265-269
        • Baloescu C.
        Diagnostic imaging in emergency medicine: how much is too much?.
        Ann Emerg Med. 2018; 72: 637-643
        • Centers for Disease Control and Prevention (CDC)
        Prevalence of coronary heart disease–United States, 2006–2010.
        MMWR Morb Mortal Wkly Rep. 2011; 60: 1377-1381
        • Joseph JW
        • Landry AM
        • Kennedy M
        • et al.
        Association of race and ethnicity with triage Emergency Severity Index scores and total visit Work Relative Value units for emergency department patients.
        JAMA Netw Open. 2022; 5e2231769

      Linked Article

      • “Machine Learning as an Adjunct to Traditional Triage in the Emergency Department”
        Journal of Emergency Medicine
        • Preview
          There are many applications of artificial intelligence (AI) in medicine, however, few clinicians have familiarity or experience with them. Machine learning, a subset of AI is the use of computers to “learn” and improve performance based on data provided without programming.1 Data is analyzed based on labeled outcomes in so-called supervised learning. Deep learning is computationally more complex and is said to mimic the human brain with interactions involving artificial neural networks. These methods suffer from a “black box” phenomenon that are poorly understood by clinicians, other than those who have a strong knowledge of computer science.
        • Full-Text
        • PDF