• Brandon C Cummings, Joseph M Blackmer, Jonathan R Motyka, Negar Farzaneh, Loc Cao, Erin L Bisco, James D Glassbrook, Michael D Roebuck, Christopher E Gillies, Andrew J Admon, Richard P Medlin, Karandeep Singh, Michael W Sjoding, Kevin R Ward, and Sardar Ansari.
• The Max Harry Weil Institute of Critical Care Research & Innovation, University of Michigan, Ann Arbor, MI.
• Crit. Care Med. 2023 Jun 1; 51 (6): 775786775-786.

ObjectivesImplementing a predictive analytic model in a new clinical environment is fraught with challenges. Dataset shifts such as differences in clinical practice, new data acquisition devices, or changes in the electronic health record (EHR) implementation mean that the input data seen by a model can differ significantly from the data it was trained on. Validating models at multiple institutions is therefore critical. Here, using retrospective data, we demonstrate how Predicting Intensive Care Transfers and other UnfoReseen Events (PICTURE), a deterioration index developed at a single academic medical center, generalizes to a second institution with significantly different patient population.DesignPICTURE is a deterioration index designed for the general ward, which uses structured EHR data such as laboratory values and vital signs.SettingThe general wards of two large hospitals, one an academic medical center and the other a community hospital.SubjectsThe model has previously been trained and validated on a cohort of 165,018 general ward encounters from a large academic medical center. Here, we apply this model to 11,083 encounters from a separate community hospital.InterventionsNone.Measurements And Main ResultsThe hospitals were found to have significant differences in missingness rates (> 5% difference in 9/52 features), deterioration rate (4.5% vs 2.5%), and racial makeup (20% non-White vs 49% non-White). Despite these differences, PICTURE's performance was consistent (area under the receiver operating characteristic curve [AUROC], 0.870; 95% CI, 0.861-0.878), area under the precision-recall curve (AUPRC, 0.298; 95% CI, 0.275-0.320) at the first hospital; AUROC 0.875 (0.851-0.902), AUPRC 0.339 (0.281-0.398) at the second. AUPRC was standardized to a 2.5% event rate. PICTURE also outperformed both the Epic Deterioration Index and the National Early Warning Score at both institutions.ConclusionsImportant differences were observed between the two institutions, including data availability and demographic makeup. PICTURE was able to identify general ward patients at risk of deterioration at both hospitals with consistent performance (AUROC and AUPRC) and compared favorably to existing metrics.Copyright © 2023 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine and Wolters Kluwer Health, Inc.

21 keywords...

hide…