Evaluating Reinforcement Learning Algorithms in Observational Health Settings

Abstract

Much attention has been devoted recently to the development of machine learning algorithms with the goal of improving treatment policies in healthcare. Reinforcement learning (RL) is a sub-field within machine learning that is concerned with learning how to make sequences of decisions so as to optimize long-term effects. Already, RL algorithms have been proposed to identify decision-making strategies for mechanical ventilation, sepsis management and treatment of schizophrenia. However, before implementing treatment policies learned by black-box algorithms in high-stakes clinical decision problems, special care must be taken in the evaluation of these policies. In this document, our goal is to expose some of the subtleties associated with evaluating RL algorithms in healthcare. We aim to provide a conceptual starting point for clinical and computational researchers to ask the right questions when designing and evaluating algorithms for new ways of treating patients. In the following, we describe how choices about how to summarize a history, variance of statistical estimators, and confounders in more ad-hoc measures can result in unreliable, even misleading estimates of the quality of a treatment policy. We also provide suggestions for mitigating these effects---for while there is much promise for mining observational health data to uncover better treatment policies, evaluation must be performed thoughtfully.

References in corpus (3)

• Towards A Rigorous Science of Interpretable Machine Learning
• Particle Swarm Optimization for Generating Interpretable Fuzzy Reinforcement Learning Policies
• Deep Reinforcement Learning for Sepsis Treatment

Cited by in corpus (30)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• A Minimalist Approach to Offline Reinforcement Learning
• Causality for Machine Learning
• Reinforcement Learning in Healthcare: A Survey
• Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• What are the Statistical Limits of Offline RL with Linear Function Approximation?
• Deconfounding Reinforcement Learning in Observational Settings
• Bridging adaptive management and reinforcement learning for more robust decisions
• Deep Reinforcement Learning for Closed-Loop Blood Glucose Control
• Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment
• Model-Based Reinforcement Learning for Sepsis Treatment
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• CoinDICE: Off-Policy Confidence Interval Estimation
• Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation
• Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions
• Is Deep Reinforcement Learning Ready for Practical Applications in Healthcare? A Sensitivity Analysis of Duel-DDQN for Hemodynamic Management in Sepsis Patients
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Challenges for Reinforcement Learning in Healthcare
• Missingness as Stability: Understanding the Structure of Missingness in Longitudinal EHR data and its Impact on Reinforcement Learning in Healthcare
• Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
• Trajectory Inspection: A Method for Iterative Clinician-Driven Design of Reinforcement Learning Studies
• Towards Robust Off-Policy Evaluation via Human Inputs
• Learning Individualized Treatment Rules with Estimated Translated Inverse Propensity Score
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluation
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• The reinforcement learning-based multi-agent cooperative approach for the adaptive speed regulation on a metallurgical pickling line
• The Impact of Time Series Length and Discretization on Longitudinal Causal Estimation Methods
• Stacking interventions for equitable outcomes