Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration

Abstract

In this paper, sample-aware policy entropy regularization is proposed to enhance the conventional policy entropy regularization for better exploration. Exploiting the sample distribution obtainable from the replay buffer, the proposed sample-aware entropy regularization maximizes the entropy of the weighted sum of the policy action distribution and the sample action distribution from the replay buffer for sample-efficient exploration. A practical algorithm named diversity actor-critic (DAC) is developed by applying policy iteration to the objective function with the proposed sample-aware entropy regularization. Numerical results show that DAC significantly outperforms existing recent algorithms for reinforcement learning.

arXiv commentAccepted to Proceedings of the 38th International Conference on Machine Learning

References in corpus (25)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• VIME: Variational Information Maximizing Exploration
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Parameter Space Noise for Exploration
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means
• Exploration by Random Network Distillation
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Sample Efficient Actor-Critic with Experience Replay
• Equivalence Between Policy Gradients and Soft Q-Learning
• Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Provably Efficient Maximum Entropy Exploration
• Combining policy gradient and Q-learning
• Diversity-Driven Exploration Strategy for Deep Reinforcement Learning
• Taming the Noise in Reinforcement Learning via Soft Updates
• Generative Adversarial Self-Imitation Learning
• Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
• Learning Self-Imitating Diverse Policies
• Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learning
• Leveraging exploration in off-policy algorithms via normalizing flows
• Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learning

Cited by in corpus (5)

• Adversarially Guided Actor-Critic
• A Reinforcement Learning Formulation of the Lyapunov Optimization: Application to Edge Computing Systems with Queue Stability
• A Max-Min Entropy Framework for Reinforcement Learning
• Learning Diverse Policies with Soft Self-Generated Guidance
• PAC-Bayesian Soft Actor-Critic Learning