EMI: Exploration with Mutual Information

Abstract

Reinforcement learning algorithms struggle when the reward signal is very sparse. In these cases, naive random exploration methods essentially rely on a random walk to stumble onto a rewarding state. Recent works utilize intrinsic motivation to guide the exploration via generative models, predictive forward models, or discriminative modeling of novelty. We propose EMI, which is an exploration method that constructs embedding representation of states and actions that does not rely on generative decoding of the full observation but extracts predictive signals that can be used to guide exploration based on forward prediction in the representation space. Our experiments show competitive results on challenging locomotion tasks with continuous control and on image-based exploration tasks with discrete actions on Atari. The source code is available at https://github.com/snu-mllab/EMI .

arXiv commentAccepted and to appear at ICML 2019

Cited by in corpus (16)

• A survey on intrinsic motivation in reinforcement learning
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• Unsupervised State Representation Learning in Atari
• Deep Reinforcement and InfoMax Learning
• Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• MADE: Exploration via Maximizing Deviation from Explored Regions
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Principled Exploration via Optimistic Bootstrapping and Backward Induction
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
• Influence-Based Multi-Agent Exploration
• DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck
• An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning
• An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewards
• Mastering Rate based Curriculum Learning