Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach

Abstract

Sepsis is a leading cause of mortality in intensive care units (ICUs) and costs hospitals billions annually. Treating a septic patient is highly challenging, because individual patients respond very differently to medical interventions and there is no universally agreed-upon treatment for sepsis. Understanding more about a patient's physiological state at a given time could hold the key to effective treatment policies. In this work, we propose a new approach to deduce optimal treatment policies for septic patients by using continuous state-space models and deep reinforcement learning. Learning treatment policies over continuous spaces is important, because we retain more of the patient's physiological information. Our model is able to learn clinically interpretable treatment policies, similar in important aspects to the treatment policies of physicians. Evaluating our algorithm on past ICU patient data, we find that our model could reduce patient mortality in the hospital by up to 3.6% over observed clinical policies, from a baseline mortality of 13.7%. The learned treatment policies could be used to aid intensive care clinicians in medical decision making and improve the likelihood of patient survival.

References in corpus (2)

• Adam: A Method for Stochastic Optimization
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

Cited by in corpus (34)

• MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III
• Reinforcement Learning in Healthcare: A Survey
• Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks
• Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling
• Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
• A Survey of Deep Learning for Scientific Discovery
• Representation and Reinforcement Learning for Personalized Glycemic Control in Septic Patients
• Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learning
• Deconfounding Reinforcement Learning in Observational Settings
• Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters
• The Faults in Our Pi Stars: Security Issues and Open Challenges in Deep Reinforcement Learning
• Exploration on Generating Traditional Chinese Medicine Prescription from Symptoms with an End-to-End method
• Representation Learning for Electronic Health Records
• Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding
• POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learning
• Precision medicine as a control problem: Using simulation and deep reinforcement learning to discover adaptive, personalized multi-cytokine therapy for sepsis
• Deep Reinforcement Learning for Clinical Decision Support: A Brief Survey
• Is Deep Reinforcement Learning Ready for Practical Applications in Healthcare? A Sensitivity Analysis of Duel-DDQN for Hemodynamic Management in Sepsis Patients
• Medical Dead-ends and Learning to Identify High-risk States and Treatments
• Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
• Entropy-Aware Model Initialization for Effective Exploration in Deep Reinforcement Learning
• RL4health: Crowdsourcing Reinforcement Learning for Knee Replacement Pathway Optimization
• Dynamic Measurement Scheduling for Event Forecasting using Deep RL
• Feature-Based Interpretable Reinforcement Learning based on State-Transition Models
• Challenges for Reinforcement Learning in Healthcare
• Identifying Distinct, Effective Treatments for Acute Hypotension with SODA-RL: Safely Optimized Diverse Accurate Reinforcement Learning
• Reinforcement Learning for Intelligent Healthcare Systems: A Comprehensive Survey
• Dynamic Measurement Scheduling for Adverse Event Forecasting using Deep RL
• Trajectory Inspection: A Method for Iterative Clinician-Driven Design of Reinforcement Learning Studies
• Learning without Knowing: Unobserved Context in Continuous Transfer Reinforcement Learning
• Medical Knowledge Integration into Reinforcement Learning Algorithms for Dynamic Treatment Regimes
• Minimalistic Attacks: How Little it Takes to Fool a Deep Reinforcement Learning Policy
• Nonparametric Additive Value Functions: Interpretable Reinforcement Learning with an Application to Surgical Recovery
• A Preliminary Approach for Learning Relational Policies for the Management of Critically Ill Children