Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
arXiv:1910.02208
Abstract
We aim to develop off-policy DRL algorithms that not only exceed state-of-the-art performance but are also simple and minimalistic. For standard continuous control benchmarks, Soft Actor-Critic (SAC), which employs entropy maximization, currently provides state-of-the-art performance. We first demonstrate that the entropy term in SAC addresses action saturation due to the bounded nature of the action spaces, with this insight, we propose a streamlined algorithm with a simple normalization scheme or with inverted gradients. We show that both approaches can match SAC's sample efficiency performance without the need of entropy maximization, we then propose a simple non-uniform sampling method for selecting transitions from the replay buffer during training. Extensive experimental results demonstrate that our proposed sampling scheme leads to state of the art sample efficiency on challenging continuous control tasks. We combine all of our findings into one simple algorithm, which we call Streamlined Off Policy with Emphasizing Recent Experience, for which we provide robust public-domain code.
References in corpus (23)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- Prioritized Experience Replay
- Soft Actor-Critic Algorithms and Applications
- Dueling Network Architectures for Deep Reinforcement Learning
- Distributed Prioritized Experience Replay
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Deep Reinforcement Learning that Matters
- Hindsight Experience Replay
- Continuous Deep Q-Learning with Model-based Acceleration
- Safe Exploration in Continuous Action Spaces
- Benchmarking Model-Based Reinforcement Learning
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- Sample Efficient Actor-Critic with Experience Replay
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
- Deep Reinforcement Learning in Parameterized Action Space
- Curiosity-Driven Experience Prioritization via Density Estimation
- Clipped Action Policy Gradient
- Marginal Policy Gradients: A Unified Family of Estimators for Bounded Action Spaces with Applications