Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPs

Abstract

Health-related data is noisy and stochastic in implying the true physiological states of patients, limiting information contained in single-moment observations for sequential clinical decision making. We model patient-clinician interactions as partially observable Markov decision processes (POMDPs) and optimize sequential treatment based on belief states inferred from history sequence. To facilitate inference, we build a variational generative model and boost state representation with a recurrent neural network (RNN), incorporating an auxiliary loss from sequence auto-encoding. Meanwhile, we optimize a continuous policy of drug levels with an actor-critic method where policy gradients are obtained from a stablized off-policy estimate of advantage function, with the value of belief state backed up by parallel best-first suffix trees. We exploit our methodology in optimizing dosages of vasopressor and intravenous fluid for sepsis patients using a retrospective intensive care dataset and evaluate the learned policy with off-policy policy evaluation (OPPE). The results demonstrate that modelling as POMDPs yields better performance than MDPs, and that incorporating heuristic search improves sample efficiency.

References in corpus (11)

• A Survey on Deep Learning in Medical Image Analysis
• Value-Function Approximations for Partially Observable Markov Decision Processes
• Distributed Prioritized Experience Replay
• Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes
• Sample Efficient Actor-Critic with Experience Replay
• Variational inference for Monte Carlo objectives
• Point-Based POMDP Algorithms: Improved Analysis and Implementation
• Safe and Efficient Off-Policy Reinforcement Learning
• On Improving Deep Reinforcement Learning for POMDPs
• Auto-Encoding Sequential Monte Carlo
• Deep Variational Reinforcement Learning for POMDPs

Cited by in corpus (7)

• Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment
• An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare
• Medical Dead-ends and Learning to Identify High-risk States and Treatments
• Offline reinforcement learning with uncertainty for treatment strategies in sepsis
• Optimal Control of Mechanical Ventilators with Learned Respiratory Dynamics
• Predicting the Need for Blood Transfusion in Intensive Care Units with Reinforcement Learning
• Medical Knowledge Integration into Reinforcement Learning Algorithms for Dynamic Treatment Regimes