Causality and Batch Reinforcement Learning: Complementary Approaches To Planning In Unknown Domains
arXiv:2006.02579
Abstract
Reinforcement learning algorithms have had tremendous successes in online learning settings. However, these successes have relied on low-stakes interactions between the algorithmic agent and its environment. In many settings where RL could be of use, such as health care and autonomous driving, the mistakes made by most online RL algorithms during early training come with unacceptable costs. These settings require developing reinforcement learning algorithms that can operate in the so-called batch setting, where the algorithms must learn from set of data that is fixed, finite, and generated from some (possibly unknown) policy. Evaluating policies different from the one that collected the data is called off-policy evaluation, and naturally poses counter-factual questions. In this project we show how off-policy evaluation and the estimation of treatment effects in causal inference are two approaches to the same problem, and compare recent progress in these two areas.
References in corpus (13)
- Continuous control with deep reinforcement learning
- Deep Reinforcement Learning: An Overview
- Domain Adaptation: Learning Bounds and Algorithms
- Distinguishing cause from effect using observational data: methods and benchmarks
- Benchmarking Batch Deep Reinforcement Learning Algorithms
- Distributional Reinforcement Learning with Quantile Regression
- Counterfactual Risk Minimization: Learning from Logged Bandit Feedback
- Two Optimal Strategies for Active Learning of Causal Models from Interventional Data
- Safe Policy Improvement with Baseline Bootstrapping
- Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models
- Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
- Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation
- Estimating Treatment Effects with Observed Confounders and Mediators