Did I do that? Blame as a means to identify controlled effects in reinforcement learning
arXiv:2106.00266
Abstract
Identifying controllable aspects of the environment has proven to be an extraordinary intrinsic motivator to reinforcement learning agents. Despite repeatedly achieving State-of-the-Art results, this approach has only been studied as a proxy to a reward-based task and has not yet been evaluated on its own. Current methods are based on action-prediction. Humans, on the other hand, assign blame to their actions to decide what they controlled. This work proposes Controlled Effect Network (CEN), an unsupervised method based on counterfactual measures of blame to identify effects on the environment controlled by the agent. CEN is evaluated in a wide range of environments showing that it can accurately identify controlled effects. Moreover, we demonstrate CEN's capabilities as intrinsic motivator by integrating it in the state-of-the-art exploration method, achieving substantially better performance than action-prediction models.
References in corpus (16)
- Dueling Network Architectures for Deep Reinforcement Learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Counterfactual Multi-Agent Policy Gradients
- Emergent Tool Use From Multi-Agent Autocurricula
- Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
- Exploration by Random Network Distillation
- Agent57: Outperforming the Atari Human Benchmark
- Never Give Up: Learning Directed Exploration Strategies
- Causal Reasoning from Meta-reinforcement Learning
- ChainerRL: A Deep Reinforcement Learning Library
- Contingency-Aware Exploration in Reinforcement Learning
- Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
- Causal Induction from Visual Observations for Goal Directed Tasks
- Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
- Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
- Disentangling causal effects for hierarchical reinforcement learning