Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance

Abstract

Recent advances in model-free deep reinforcement learning (DRL) show that simple model-free methods can be highly effective in challenging high-dimensional continuous control tasks. In particular, Truncated Quantile Critics (TQC) achieves state-of-the-art asymptotic training performance on the MuJoCo benchmark with a distributional representation of critics; and Randomized Ensemble Double Q-Learning (REDQ) achieves high sample efficiency that is competitive with state-of-the-art model-based methods using a high update-to-data ratio and target randomization. In this paper, we propose a novel model-free algorithm, Aggressive Q-Learning with Ensembles (AQE), which improves the sample-efficiency performance of REDQ and the asymptotic performance of TQC, thereby providing overall state-of-the-art performance during all stages of training. Moreover, AQE is very simple, requiring neither distributional representation of critics nor target randomization. The effectiveness of AQE is further supported by our extensive experiments, ablations, and theoretical results.

References in corpus (10)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• Deep Reinforcement Learning that Matters
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Maxmin Q-learning: Controlling the Estimation Bias of Q-learning