Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage

Abstract

We study model-based offline Reinforcement Learning with general function approximation without a full coverage assumption on the offline data distribution. We present an algorithm named Constrained Pessimistic Policy Optimization (CPPO)which leverages a general function class and uses a constraint over the model class to encode pessimism. Under the assumption that the ground truth model belongs to our function class (i.e., realizability in the function class), CPPO has a PAC guarantee with offline data only providing partial coverage, i.e., it can learn a policy that competes against any policy that is covered by the offline data. We then demonstrate that this algorithmic framework can be applied to many specialized Markov Decision Processes where additional structural assumptions can further refine the concept of partial coverage. Two notable examples are: (1) low-rank MDP with representation learning where the partial coverage condition is defined using a relative condition number measured by the unknown ground truth feature representation; (2) factored MDP where the partial coverage condition is defined using density ratio based concentrability coefficients associated with individual factors.

We changed the title from the first version. This is a longer version of the article accepted in ICLR 2022. The following things are added (1) a new algorithm CPPO-LR where the constraint is given in a log-likelihood form, (2) how to instantiate CPPO on (nonparametric) linear MDPs, (3) posterior sampling in a model-free way

References in corpus (31)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Policy Iteration for Factored MDPs
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• GenDICE: Generalized Offline Estimation of Stationary Values
• Information-Theoretic Considerations in Batch Reinforcement Learning
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• What are the Statistical Limits of Offline RL with Linear Function Approximation?
• FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs
• Batch Value-function Approximation with Only Realizability
• PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning
• Information Theoretic Regret Bounds for Online Nonlinear Control
• Is Pessimism Provably Efficient for Offline RL?
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Toward the Fundamental Limits of Imitation Learning
• Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
• Bellman-consistent Pessimism for Offline Reinforcement Learning
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
• Model-free Representation Learning and Exploration in Low-rank MDPs
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• Leveraging Good Representations in Linear Contextual Bandits
• Stable Policy Optimization via Off-Policy Divergence Regularization
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Corruption-Robust Offline Reinforcement Learning
• Continuous Doubly Constrained Batch Reinforcement Learning
• Learning Good State and Action Representations via Tensor Decomposition
• Provably Efficient Representation Selection in Low-rank Markov Decision Processes: From Online to Offline RL