Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience

Abstract

Soft Actor-Critic (SAC) is an off-policy actor-critic reinforcement learning algorithm, essentially based on entropy regularization. SAC trains a policy by maximizing the trade-off between expected return and entropy (randomness in the policy). It has achieved state-of-the-art performance on a range of continuous-control benchmark tasks, outperforming prior on-policy and off-policy methods. SAC works in an off-policy fashion where data are sampled uniformly from past experiences (stored in a buffer) using which parameters of the policy and value function networks are updated. We propose certain crucial modifications for boosting the performance of SAC and make it more sample efficient. In our proposed improved SAC, we firstly introduce a new prioritization scheme for selecting better samples from the experience replay buffer. Secondly we use a mixture of the prioritized off-policy data with the latest on-policy data for training the policy and the value function networks. We compare our approach with the vanilla SAC and some recent variants of SAC and show that our approach outperforms the said algorithmic benchmarks. It is comparatively more stable and sample efficient when tested on a number of continuous control tasks in MuJoCo environments.

References in corpus (14)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Dueling Network Architectures for Deep Reinforcement Learning
• Distributed Prioritized Experience Replay
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Sample Efficient Actor-Critic with Experience Replay
• Equivalence Between Policy Gradients and Soft Q-Learning
• Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past
• Prioritized Sequence Experience Replay
• Experience Replay with Likelihood-free Importance Weights
• Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies
• Reconciling $\lambda$-Returns with Experience Replay
• Band-limited Soft Actor Critic Model