AutoScore-Imbalance: An interpretable machine learning tool for development of clinical scores with rare events data

Abstract

Background: Medical decision-making impacts both individual and public health. Clinical scores are commonly used among a wide variety of decision-making models for determining the degree of disease deterioration at the bedside. AutoScore was proposed as a useful clinical score generator based on machine learning and a generalized linear model. Its current framework, however, still leaves room for improvement when addressing unbalanced data of rare events. Methods: Using machine intelligence approaches, we developed AutoScore-Imbalance, which comprises three components: training dataset optimization, sample weight optimization, and adjusted AutoScore. All scoring models were evaluated on the basis of their area under the curve (AUC) in the receiver operating characteristic analysis and balanced accuracy (i.e., mean value of sensitivity and specificity). By utilizing a publicly accessible dataset from Beth Israel Deaconess Medical Center, we assessed the proposed model and baseline approaches in the prediction of inpatient mortality. Results: AutoScore-Imbalance outperformed baselines in terms of AUC and balanced accuracy. The nine-variable AutoScore-Imbalance sub-model achieved the highest AUC of 0.786 (0.732-0.839) while the eleven-variable original AutoScore obtained an AUC of 0.723 (0.663-0.783), and the logistic regression with 21 variables obtained an AUC of 0.743 (0.685-0.800). The AutoScore-Imbalance sub-model (using down-sampling algorithm) yielded an AUC of 0. 0.771 (0.718-0.823) with only five variables, demonstrating a good balance between performance and variable sparsity. Conclusions: The AutoScore-Imbalance tool has the potential to be applied to highly unbalanced datasets to gain further insight into rare medical events and to facilitate real-world clinical decision-making.

References in corpus (9)

• SMOTE: Synthetic Minority Over-sampling Technique
• A Unified Approach to Interpreting Model Predictions
• Generative Adversarial Networks: An Overview
• Towards A Rigorous Science of Interpretable Machine Learning
• Class Imbalance Problem in Data Mining Review
• Modeling Tabular data using Conditional GAN
• Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead
• AutoScore-Survival: Developing interpretable machine learning-based time-to-event scores with right-censored survival data
• Interactivity and Transparency in Medical Risk Assessment with Supersparse Linear Integer Models

Cited by in corpus (2)

• Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies
• Benchmarking emergency department triage prediction models with machine learning and large public electronic health records