Bellman-consistent Pessimism for Offline Reinforcement Learning

Abstract

The use of pessimism, when reasoning about datasets lacking exhaustive exploration has recently gained prominence in offline reinforcement learning. Despite the robustness it adds to the algorithm, overly pessimistic reasoning can be equally damaging in precluding the discovery of good policies, which is an issue for the popular bonus-based pessimism. In this paper, we introduce the notion of Bellman-consistent pessimism for general function approximation: instead of calculating a point-wise lower bound for the value function, we implement pessimism at the initial state over the set of functions consistent with the Bellman equations. Our theoretical guarantees only require Bellman closedness as standard in the exploratory setting, in which case bonus-based pessimism fails to provide guarantees. Even in the special case of linear function approximation where stronger expressivity assumptions hold, our result improves upon a recent bonus-based approach by $\mathcal{O}(d)$ in its sample complexity when the action space is finite. Remarkably, our algorithms automatically adapt to the best bias-variance tradeoff in the hindsight, whereas most prior approaches require tuning extra hyperparameters a priori.

NeurIPS 2021 (Oral)

References in corpus (8)

• Conservative Q-Learning for Offline Reinforcement Learning
• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• A Theory of Regularized Markov Decision Processes
• Information-Theoretic Considerations in Batch Reinforcement Learning
• Approximate Policy Iteration Schemes: A Comparison
• Provably Good Batch Reinforcement Learning Without Great Exploration
• Statistical Linear Estimation with Penalized Estimators: an Application to Reinforcement Learning
• Instabilities of Offline RL with Pre-Trained Neural Representation

Cited by in corpus (5)

• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• Representation Learning for Online and Offline RL in Low-rank MDPs
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• Pessimistic Off-Policy Optimization for Learning to Rank