The Importance of Pessimism in Fixed-Dataset Policy Optimization

Abstract

We study worst-case guarantees on the expected return of fixed-dataset policy optimization algorithms. Our core contribution is a unified conceptual and mathematical framework for the study of algorithms in this regime. This analysis reveals that for naive approaches, the possibility of erroneous value overestimation leads to a difficult-to-satisfy requirement: in order to guarantee that we select a policy which is near-optimal, we may need the dataset to be informative of the value of every policy. To avoid this, algorithms can follow the pessimism principle, which states that we should choose the policy which acts optimally in the worst possible world. We show why pessimistic algorithms can achieve good performance even when the dataset is not informative of every policy, and derive families of algorithms which follow this principle. These theoretical findings are validated by experiments on a tabular gridworld, and deep learning experiments on four MinAtar environments.

References in corpus (16)

• Trust Region Policy Optimization
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Empirical Bernstein Bounds and Sample Variance Penalization
• MOReL : Model-Based Offline Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Critic Regularized Regression
• Revisiting Fundamentals of Experience Replay
• Safe Policy Improvement with Baseline Bootstrapping
• Safe Policy Improvement by Minimizing Robust Baseline Regret
• MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments
• Provably Good Batch Reinforcement Learning Without Great Exploration
• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization

Cited by in corpus (14)

• A Minimalist Approach to Offline Reinforcement Learning
• Is Pessimism Provably Efficient for Offline RL?
• Value Penalized Q-Learning for Recommender Systems
• CoinDICE: Off-Policy Confidence Interval Estimation
• Offline RL Without Off-Policy Evaluation
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Corruption-Robust Offline Reinforcement Learning
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• On the Optimality of Batch Policy Optimization Algorithms
• You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
• Offline RL With Resource Constrained Online Deployment
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation