Mutual Information State Intrinsic Control
arXiv:2103.08107
Abstract
Reinforcement learning has been shown to be highly successful at many challenging tasks. However, success heavily relies on well-shaped rewards. Intrinsically motivated RL attempts to remove this constraint by defining an intrinsic reward function. Motivated by the self-consciousness concept in psychology, we make a natural assumption that the agent knows what constitutes itself, and propose a new intrinsic objective that encourages the agent to have maximum control on the environment. We mathematically formalize this reward as the mutual information between the agent state and the surrounding state under the current agent policy. With this new intrinsic motivation, we are able to outperform previous methods, including being able to complete the pick-and-place task for the first time without using any task reward. A video showing experimental results is available at https://youtu.be/AUCwc9RThpk.
arXiv commentPublished in International Conference on Learning Representations (ICLR 2021) as Spotlight (top 5%), Link: https://openreview.net/forum?id=OthEq8I5v1. arXiv admin note: text overlap with arXiv:2002.01963
References in corpus (14)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Estimating Mutual Information
- Learning deep representations by mutual information estimation and maximization
- Dota 2 with Large Scale Deep Reinforcement Learning
- MINE: Mutual Information Neural Estimation
- Large-Scale Study of Curiosity-Driven Learning
- Deep Variational Information Bottleneck
- Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- Variational Intrinsic Control
- Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning
- Relevant sparse codes with variational information bottleneck
- Maximum Entropy-Regularized Multi-Goal Reinforcement Learning
- Unsupervised Control Through Non-Parametric Discriminative Rewards