Causally Correct Partial Models for Reinforcement Learning
arXiv:2002.02836
Abstract
In reinforcement learning, we can learn a model of future observations and rewards, and use it to plan the agent's next actions. However, jointly modeling future observations can be computationally expensive or even intractable if the observations are high-dimensional (e.g. images). For this reason, previous works have considered partial models, which model only part of the observation. In this paper, we show that partial models can be causally incorrect: they are confounded by the observations they don't model, and can therefore lead to incorrect planning. To address this, we introduce a general family of partial models that are provably causally correct, yet remain fast because they do not need to fully model future observations.
References in corpus (10)
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Model-Based Reinforcement Learning for Atari
- Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
- Recurrent Environment Simulators
- Causal Discovery with Reinforcement Learning
- Recurrent Reinforcement Learning: A Hybrid Approach
- Neural Predictive Belief Representations
- Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
- Deconfounding Reinforcement Learning in Observational Settings
- Combating the Compounding-Error Problem with a Multi-step Model
Cited by in corpus (8)
- Muesli: Combining Improvements in Policy Optimization
- Counterfactual Data Augmentation using Locally Factored Dynamics
- Causal Influence Detection for Improving Efficiency in Reinforcement Learning
- Shaking the foundations: delusions in sequence models for interaction and control
- Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
- Procedural Generalization by Planning with Self-Supervised World Models
- Planning from Pixels using Inverse Dynamics Models
- Self-Consistent Models and Values