Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics

Abstract

The overestimation bias is one of the major impediments to accurate off-policy learning. This paper investigates a novel way to alleviate the overestimation bias in a continuous control setting. Our method---Truncated Quantile Critics, TQC,---blends three ideas: distributional representation of a critic, truncation of critics prediction, and ensembling of multiple critics. Distributional representation and truncation allow for arbitrary granular overestimation control, while ensembling provides additional score improvements. TQC outperforms the current state of the art on all environments from the continuous control benchmark suite, demonstrating 25% improvement on the most challenging Humanoid environment.

arXiv commentUnder review by the International Conference on Machine Learning

References in corpus (15)

• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Distributed Distributional Deterministic Policy Gradients
• Implicit Quantile Networks for Distributional Reinforcement Learning
• A Distributional Perspective on Reinforcement Learning
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• Simple random search provides a competitive approach to reinforcement learning
• Distributional Reinforcement Learning with Quantile Regression
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Fully Parameterized Quantile Function for Distributional Reinforcement Learning
• Estimating the Maximum Expected Value: An Analysis of (Nested) Cross Validation and the Maximum Sample Average
• Sequential Test for the Lowest Mean: From Thompson to Murphy Sampling

Cited by in corpus (8)

• Learning to Grasp on the Moon from 3D Octree Observations with Deep Reinforcement Learning
• The Role of Morphological Variation in Evolutionary Robotics: Maximizing Performance and Robustness
• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
• GMAC: A Distributional Perspective on Actor-Critic Framework
• Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration
• Direct Random Search for Fine Tuning of Deep Reinforcement Learning Policies
• Adaptation of Quadruped Robot Locomotion with Meta-Learning