Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learning
arXiv:2002.01963
Abstract
In reinforcement learning, an agent learns to reach a set of goals by means of an external reward signal. In the natural world, intelligent organisms learn from internal drives, bypassing the need for external signals, which is beneficial for a wide range of tasks. Motivated by this observation, we propose to formulate an intrinsic objective as the mutual information between the goal states and the controllable states. This objective encourages the agent to take control of its environment. Subsequently, we derive a surrogate objective of the proposed reward function, which can be optimized efficiently. Lastly, we evaluate the developed framework in different robotic manipulation and navigation tasks and demonstrate the efficacy of our approach. A video showing experimental results is available at https://youtu.be/CT4CKMWBYz0
arXiv comment15 pages, 8 figures
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Estimating Mutual Information
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Learning deep representations by mutual information estimation and maximization
- VIME: Variational Information Maximizing Exploration
- Large-Scale Study of Curiosity-Driven Learning
- Hindsight Experience Replay
- Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- Variational Intrinsic Control
- Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
- Maximum Entropy-Regularized Multi-Goal Reinforcement Learning
- Curiosity-Driven Experience Prioritization via Density Estimation
- Energy-Based Hindsight Experience Prioritization