Predicting the Need for Blood Transfusion in Intensive Care Units with Reinforcement Learning

Abstract

As critically ill patients frequently develop anemia or coagulopathy, transfusion of blood products is a frequent intervention in the Intensive Care Units (ICU). However, inappropriate transfusion decisions made by physicians are often associated with increased risk of complications and higher hospital costs. In this work, we aim to develop a decision support tool that uses available patient information for transfusion decision-making on three common blood products (red blood cells, platelets, and fresh frozen plasma). To this end, we adopt an off-policy batch reinforcement learning (RL) algorithm, namely, discretized Batch Constrained Q-learning, to determine the best action (transfusion or not) given observed patient trajectories. Simultaneously, we consider different state representation approaches and reward design mechanisms to evaluate their impacts on policy learning. Experiments are conducted on two real-world critical care datasets: the MIMIC-III and the UCSF. Results demonstrate that policy recommendations on transfusion achieved comparable matching against true hospital policies via accuracy and weighted importance sampling evaluations on the MIMIC-III dataset. Furthermore, a combination of transfer learning (TL) and RL on the data-scarce UCSF dataset can provide up to $17.02% improvement in terms of accuracy, and up to 18.94% and 21.63% improvement in jump-start and asymptotic performance in terms of weighted importance sampling averaged over three transfusion tasks. Finally, simulations on transfusion decisions suggest that the transferred RL policy could reduce patients' estimated 28-day mortality rate by 2.74% and decreased acuity rate by 1.18% on the UCSF dataset.

Proceedings of 13th International ACM Conference on Bioinformatics, Computational Biology and Health Informatics

References in corpus (6)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Applications of Deep Learning and Reinforcement Learning to Biological Data
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Q-learning with censored data
• BERTSurv: BERT-Based Survival Models for Predicting Outcomes of Trauma Patients
• Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPs