Purpose: Mortality risk prediction helps clinicians make better decisions in patient healthcare. However, existing severity scoring systems or algorithms used in intensive care units (ICUs) often rely on laborious manual collection of complex variables and lack sufficient validation in diverse clinical environments, thus limiting their practical applicability. This study aims to evaluate the performance of machine learning models that utilize routinely collected clinical data for short-term mortality risk prediction.
Patients and Methods: Using the eICU Collaborative Research Database, we identified a cohort of 12,393 ICU patients, who were randomly divided into a training group and a validation group at a ratio of 9:1. The models utilized routine variables obtained from regular medical workflows, including age, gender, physiological measurements, and usage of vasoactive medications within a 24-hour period prior to patient discharge. Four different machine learning algorithms, namely logistic regression, random forest, extreme gradient boosting (XGboost), and artificial neural network were employed to develop the mortality risk prediction model. We compared the discrimination and calibration performance of these models in assessing mortality risk within 1-week time window.
Results: Among the tested models, the XGBoost algorithm demonstrated the highest performance, with an area under the receiver operating characteristic curve (AUROC) of 0.9702, an area under precision and recall curves (AUPRC) of 0.8517, and a favorable Brier score of 0.0259 for 24-hour mortality risk prediction. Although the model’s performance decreased when considering larger time windows, it still achieved a comparable AUROC of 0.9184 and AUPRC of 0.5519 for 3-day mortality risk prediction.
Conclusion: The findings demonstrate the feasibility of developing a highly accurate and well-calibrated model based on the XGBoost algorithm for short-term mortality risk prediction with easily accessible and interpretative data. These results enhance confidence in the application of the machine learning model to clinical practice.
Keywords: intensive care unit, XGBoost, routinely collected data, short-term mortality risk