An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare

Abstract

Reinforcement Learning (RL) has recently been applied to sequential estimation and prediction problems identifying and developing hypothetical treatment strategies for septic patients, with a particular focus on offline learning with observational data. In practice, successful RL relies on informative latent states derived from sequential observations to develop optimal treatment strategies. To date, how best to construct such states in a healthcare setting is an open question. In this paper, we perform an empirical study of several information encoding architectures using data from septic patients in the MIMIC-III dataset to form representations of a patient state. We evaluate the impact of representation dimension, correlations with established acuity scores, and the treatment policies derived from them. We find that sequentially formed state representations facilitate effective policy learning in batch settings, validating a more thoughtful approach to representation learning that remains faithful to the sequential and partial nature of healthcare data.

arXiv commentTo appear in proceedings of the 2020 Machine Learning for Health workshop at NeurIPS

References in corpus (12)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Playing Atari with Deep Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Reinforcement Learning in Healthcare: A Survey
• Deep Reinforcement Learning for Sepsis Treatment
• Decoupling Dynamics and Reward for Transfer Learning
• Model-Based Reinforcement Learning for Sepsis Treatment
• The Actor Search Tree Critic (ASTC) for Off-Policy POMDP Learning in Medical Decision Making
• Representation Learning for Electronic Health Records
• Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPs
• Representation Learning with Autoencoders for Electronic Health Records: A Comparative Study
• Optimizing Medical Treatment for Sepsis in Intensive Care: from Reinforcement Learning to Pre-Trial Evaluation

Cited by in corpus (2)

• Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning