A Max-Min Entropy Framework for Reinforcement Learning
arXiv:2106.10517
Abstract
In this paper, we propose a max-min entropy framework for reinforcement learning (RL) to overcome the limitation of the soft actor-critic (SAC) algorithm implementing the maximum entropy RL in model-free sample-based learning. Whereas the maximum entropy RL guides learning for policies to reach states with high entropy in the future, the proposed max-min entropy framework aims to learn to visit states with low entropy and maximize the entropy of these low-entropy states to promote better exploration. For general Markov decision processes (MDPs), an efficient algorithm is constructed under the proposed max-min entropy framework based on disentanglement of exploration and exploitation. Numerical results show that the proposed algorithm yields drastic performance improvement over the current state-of-the-art RL algorithms.
arXiv commentAccepted to NeurIPS 2021
References in corpus (22)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- Soft Actor-Critic Algorithms and Applications
- VIME: Variational Information Maximizing Exploration
- Exploration by Random Network Distillation
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- Equivalence Between Policy Gradients and Soft Q-Learning
- Optimization, Learning, and Games with Predictable Sequences
- Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
- Provably Efficient Maximum Entropy Exploration
- Combining policy gradient and Q-learning
- Diversity-Driven Exploration Strategy for Deep Reinforcement Learning
- Contingency-Aware Exploration in Reinforcement Learning
- Generative Adversarial Self-Imitation Learning
- Maximum Entropy RL (Provably) Solves Some Robust RL Problems
- Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
- Frank-Wolfe Algorithms for Saddle Point Problems
- Rényi Fair Inference
- Leveraging exploration in off-policy algorithms via normalizing flows
- Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
- MULEX: Disentangling Exploitation from Exploration in Deep RL