Causal Influence Detection for Improving Efficiency in Reinforcement Learning
arXiv:2106.03443
Abstract
Many reinforcement learning (RL) environments consist of independent entities that interact sparsely. In such environments, RL agents have only limited influence over other entities in any particular situation. Our idea in this work is that learning can be efficiently guided by knowing when and what the agent can influence with its actions. To achieve this, we introduce a measure of \emph{situation-dependent causal influence} based on conditional mutual information and show that it can reliably detect states of influence. We then propose several ways to integrate this measure into RL algorithms to improve exploration and off-policy learning. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.
arXiv commentNeurIPS 2021 camera-ready version. Code available at http://github.com/martius-lab/cid-in-rl
References in corpus (19)
- Continuous control with deep reinforcement learning
- Measuring Information Transfer
- Estimating Mutual Information
- Spectral Normalization for Generative Adversarial Networks
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- VIME: Variational Information Maximizing Exploration
- Causal Effect Inference with Deep Latent-Variable Models
- Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
- Information-theoretic approach to the study of control systems
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- On Variational Bounds of Mutual Information
- Learning Independent Causal Mechanisms
- Information driven self-organization of complex robotic behaviors
- Identifiability of Causal Graphs using Functional Models
- Contingency-Aware Exploration in Reinforcement Learning
- Learning Neural Causal Models from Unknown Interventions
- Deconfounding Reinforcement Learning in Observational Settings
- Causally Correct Partial Models for Reinforcement Learning
- Mutual Information State Intrinsic Control