EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL

Abstract

Off-policy reinforcement learning holds the promise of sample-efficient learning of decision-making policies by leveraging past experience. However, in the offline RL setting -- where a fixed collection of interactions are provided and no further interactions are allowed -- it has been shown that standard off-policy RL methods can significantly underperform. Recently proposed methods often aim to address this shortcoming by constraining learned policies to remain close to the given dataset of interactions. In this work, we closely investigate an important simplification of BCQ -- a prior approach for offline RL -- which removes a heuristic design choice and naturally restricts extracted policies to remain exactly within the support of a given behavior policy. Importantly, in contrast to their original theoretical considerations, we derive this simplified algorithm through the introduction of a novel backup operator, Expected-Max Q-Learning (EMaQ), which is more closely related to the resulting practical algorithm. Specifically, in addition to the distribution support, EMaQ explicitly considers the number of samples and the proposal distribution, allowing us to derive new sub-optimality bounds which can serve as a novel measure of complexity for offline RL problems. In the offline RL setting -- the main focus of this work -- EMaQ matches and outperforms prior state-of-the-art in the D4RL benchmarks. In the online RL setting, we demonstrate that EMaQ is competitive with Soft Actor Critic. The key contributions of our empirical findings are demonstrating the importance of careful generative model design for estimating behavior policies, and an intuitive notion of complexity for offline RL problems. With its simple interpretation and fewer moving parts, such as no explicit function approximator representing the policy, EMaQ serves as a strong yet easy to implement baseline for future work.

References in corpus (22)

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Addressing Function Approximation Error in Actor-Critic Methods
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Conservative Q-Learning for Offline Reinforcement Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• MADE: Masked Autoencoder for Distribution Estimation
• Continuous Deep Q-Learning with Model-based Acceleration
• Behavior Regularized Offline Reinforcement Learning
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Input Convex Neural Networks
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• UCB Exploration via Q-Ensembles
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Q-Learning in enormous action spaces via amortized approximate maximization