An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learning

Abstract

Policy-based reinforcement learning methods suffer from the policy collapse problem. We find valued-based reinforcement learning methods with ε-greedy mechanism are capable of enjoying three characteristics, Closed-form Diversity, Objective-invariant Exploration and Adaptive Trade-off, which help value-based methods avoid the policy collapse problem. However, there does not exist a parallel mechanism for policy-based methods that achieves all three characteristics. In this paper, we propose an entropy regularization free mechanism that is designed for policy-based methods, which achieves Closed-form Diversity, Objective-invariant Exploration and Adaptive Trade-off. Our experiments show that our mechanism is super sample-efficient for policy-based methods and boosts a policy-based baseline to a new State-Of-The-Art on Arcade Learning Environment.

arXiv commentarXiv admin note: text overlap with arXiv:2105.03923

References in corpus (11)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• Dueling Network Architectures for Deep Reinforcement Learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Agent57: Outperforming the Atari Human Benchmark
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• Revisiting the Softmax Bellman Operator: New Benefits and New Perspective
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
• The Value Function Polytope in Reinforcement Learning