Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning

Abstract

Off-policy deep reinforcement learning algorithms commonly compensate for overestimation bias during temporal-difference learning by utilizing pessimistic estimates of the expected target returns. In this work, we propose Generalized Pessimism Learning (GPL), a strategy employing a novel learnable penalty to enact such pessimism. In particular, we propose to learn this penalty alongside the critic with dual TD-learning, a new procedure to estimate and minimize the magnitude of the target returns bias with trivial computational cost. GPL enables us to accurately counteract overestimation bias throughout training without incurring the downsides of overly pessimistic targets. By integrating GPL with popular off-policy algorithms, we achieve state-of-the-art results in both competitive proprioceptive and pixel-based benchmarks.

Oral at the 37th AAAI Conference on Artificial Intelligence (AAAI-23)

References in corpus (13)

• Soft Actor-Critic Algorithms and Applications
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• On Layer Normalization in the Transformer Architecture
• Exploring Model-based Planning with Policy Networks
• Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
• Discovering Reinforcement Learning Algorithms
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Better Exploration with Optimistic Actor-Critic
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• Evolving Reinforcement Learning Algorithms