Exploration by Maximizing Rényi Entropy for Reward-Free RL Framework
arXiv:2006.06193
Abstract
Exploration is essential for reinforcement learning (RL). To face the challenges of exploration, we consider a reward-free RL framework that completely separates exploration from exploitation and brings new challenges for exploration algorithms. In the exploration phase, the agent learns an exploratory policy by interacting with a reward-free environment and collects a dataset of transitions by executing the policy. In the planning phase, the agent computes a good policy for any reward function based on the dataset without further interacting with the environment. This framework is suitable for the meta RL setting where there are many reward functions of interest. In the exploration phase, we propose to maximize the Renyi entropy over the state-action space and justify this objective theoretically. The success of using Renyi entropy as the objective results from its encouragement to explore the hard-to-reach state-actions. We further deduce a policy gradient formulation for this objective and design a practical exploration algorithm that can deal with complex environments. In the planning phase, we solve for good policies given arbitrary reward functions using a batch RL algorithm. Empirically, we show that our exploration algorithm is effective and sample efficient, and results in superior policies for arbitrary reward functions in the planning phase.
Accepted by AAAI-21
References in corpus (18)
- Auto-Encoding Variational Bayes
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- VIME: Variational Information Maximizing Exploration
- Reinforcement Learning with Deep Energy-Based Policies
- Learning to reinforcement learn
- Count-Based Exploration with Neural Density Models
- Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
- Maximum a Posteriori Policy Optimisation
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- Equivalence Between Policy Gradients and Soft Q-Learning
- Benchmarking Batch Deep Reinforcement Learning Algorithms
- Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learning
- Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
- Provably Efficient Maximum Entropy Exploration
- Unsupervised Meta-Learning for Reinforcement Learning
- Reward-Free Exploration for Reinforcement Learning
- Entropy Regularization with Discounted Future State Distribution in Policy Gradient Methods