Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search

Abstract

Learning policies on data synthesized by models can in principle quench the thirst of reinforcement learning algorithms for large amounts of real experience, which is often costly to acquire. However, simulating plausible experience de novo is a hard problem for many complex environments, often resulting in biases for model-based policy evaluation and search. Instead of de novo synthesis of data, here we assume logged, real experience and model alternative outcomes of this experience under counterfactual actions, actions that were not actually taken. Based on this, we propose the Counterfactually-Guided Policy Search (CF-GPS) algorithm for learning policies in POMDPs from off-policy experience. It leverages structural causal models for counterfactual evaluation of arbitrary policies on individual off-policy episodes. CF-GPS can improve on vanilla model-based RL algorithms by making use of available logged data to de-bias model predictions. In contrast to off-policy algorithms based on Importance Sampling which re-weight data, CF-GPS leverages a model to explicitly consider alternative outcomes, allowing the algorithm to make better use of experience data. We find empirically that these advantages translate into improved policy evaluation and search results on a non-trivial grid-world task. Finally, we show that CF-GPS generalizes the previously proposed Guided Policy Search and that reparameterization-based algorithms such Stochastic Value Gradient can be interpreted as counterfactual methods.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• DRAW: A Recurrent Neural Network For Image Generation
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
• A comparative study of counterfactual estimators

Cited by in corpus (23)

• Causality for Machine Learning
• Causal Induction from Visual Observations for Goal Directed Tasks
• Deconfounding Reinforcement Learning in Observational Settings
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation
• Causally Correct Partial Models for Reinforcement Learning
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data
• Synthetic Returns for Long-Term Credit Assignment
• Causality and Batch Reinforcement Learning: Complementary Approaches To Planning In Unknown Domains
• Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
• Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
• Dynamic Causal Bayesian Optimization
• Counterfactual Explanations in Sequential Decision Making Under Uncertainty
• Resolving Spurious Correlations in Causal Models of Environments via Interventions
• Disentangling causal effects for hierarchical reinforcement learning
• Counterfactual Policy Evaluation for Decision-Making in Autonomous Driving
• Direct Policy Gradients: Direct Optimization of Policies in Discrete Action Spaces
• Value-driven Hindsight Modelling
• Counterfactual Off-Policy Training for Neural Response Generation
• Independent mechanism analysis, a new concept?
• Learning Generalized Gumbel-max Causal Mechanisms
• Importance of using appropriate baselines for evaluation of data-efficiency in deep reinforcement learning for Atari
• Did I do that? Blame as a means to identify controlled effects in reinforcement learning