Geometric Entropic Exploration

Abstract

Exploration is essential for solving complex Reinforcement Learning (RL) tasks. Maximum State-Visitation Entropy (MSVE) formulates the exploration problem as a well-defined policy optimization problem whose solution aims at visiting all states as uniformly as possible. This is in contrast to standard uncertainty-based approaches where exploration is transient and eventually vanishes. However, existing approaches to MSVE are theoretically justified only for discrete state-spaces as they are oblivious to the geometry of continuous domains. We address this challenge by introducing Geometric Entropy Maximisation (GEM), a new algorithm that maximises the geometry-aware Shannon entropy of state-visits in both discrete and continuous domains. Our key theoretical contribution is casting geometry-aware MSVE exploration as a tractable problem of optimising a simple and novel noise-contrastive objective function. In our experiments, we show the efficiency of GEM in solving several RL problems with sparse rewards, compared against other deep RL exploration approaches.

References in corpus (17)

• Continuous control with deep reinforcement learning
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• State Representation Learning for Control: An Overview
• Exploration by Random Network Distillation
• Optimizing the Latent Space of Generative Networks
• Episodic Curiosity through Reachability
• Variational Intrinsic Control
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Provably Efficient Maximum Entropy Exploration
• Efficient Exploration via State Marginal Matching
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Minimax Regret Bounds for Reinforcement Learning
• Reward-Free Exploration for Reinforcement Learning
• Autonomous exploration for navigating in non-stationary CMPs
• Marginalized State Distribution Entropy Regularization in Policy Optimization

Cited by in corpus (3)

• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Action Redundancy in Reinforcement Learning
• More Efficient Exploration with Symbolic Priors on Action Sequence Equivalences