Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation

Abstract

Reinforcement learning (RL) algorithms usually require a substantial amount of interaction data and perform well only for specific tasks in a fixed environment. In some scenarios such as healthcare, however, usually only few records are available for each patient, and patients may show different responses to the same treatment, impeding the application of current RL algorithms to learn optimal policies. To address the issues of mechanism heterogeneity and related data scarcity, we propose a data-efficient RL algorithm that exploits structural causal models (SCMs) to model the state dynamics, which are estimated by leveraging both commonalities and differences across subjects. The learned SCM enables us to counterfactually reason what would have happened had another treatment been taken. It helps avoid real (possibly risky) exploration and mitigates the issue that limited experiences lead to biased policies. We propose counterfactual RL algorithms to learn both population-level and individual-level policies. We show that counterfactual outcomes are identifiable under mild conditions and that Q- learning on the counterfactual-based augmented data set converges to the optimal value function. Experimental results on synthetic and real-world data demonstrate the efficacy of the proposed approach.

Neural Information Processing Systems Workshop on Offline Reinforcement Learning

References in corpus (13)

• Playing Atari with Deep Reinforcement Learning
• A Simple Neural Attentive Meta-Learner
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• On the Identifiability of the Post-Nonlinear Causal Model
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Evaluating Reinforcement Learning Algorithms in Observational Health Settings
• Deep Reinforcement Learning for Sepsis Treatment
• Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning
• Precision medicine as a control problem: Using simulation and deep reinforcement learning to discover adaptive, personalized multi-cytokine therapy for sepsis

Cited by in corpus (4)

• Counterfactual experience augmented off-policy reinforcement learning
• Learning Generalized Gumbel-max Causal Mechanisms
• Independent mechanism analysis, a new concept?
• Causal Learning for Socially Responsible AI