Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
arXiv:2108.08812
Abstract
Actor-critic methods are widely used in offline reinforcement learning practice, but are not so well-understood theoretically. We propose a new offline actor-critic algorithm that naturally incorporates the pessimism principle, leading to several key advantages compared to the state of the art. The algorithm can operate when the Bellman evaluation operator is closed with respect to the action value function of the actor's policies; this is a more general setting than the low-rank MDP model. Despite the added generality, the procedure is computationally tractable as it involves the solution of a sequence of second-order programs. We prove an upper bound on the suboptimality gap of the policy returned by the procedure that depends on the data coverage of any arbitrary, possibly data dependent comparator policy. The achievable guarantee is complemented with a minimax lower bound that is matching up to logarithmic factors.
arXiv commentInitial submission; appeared as spotlight talk in ICML 2021 Workshop on Theory of RL
References in corpus (41)
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Conservative Q-Learning for Offline Reinforcement Learning
- Reinforcement Learning with Deep Energy-Based Policies
- Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
- Learning Continuous Control Policies by Stochastic Value Gradients
- Behavior Regularized Offline Reinforcement Learning
- MOPO: Model-based Offline Policy Optimization
- Provably Efficient Reinforcement Learning with Linear Function Approximation
- MOReL : Model-Based Offline Reinforcement Learning
- Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
- Critic Regularized Regression
- Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
- A Theory of Regularized Markov Decision Processes
- Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
- AlgaeDICE: Policy Gradient from Arbitrary Experience
- Safe Policy Improvement with Baseline Bootstrapping
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
- Variational Policy Gradient Method for Reinforcement Learning with General Utilities
- FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs
- Provably Good Batch Reinforcement Learning Without Great Exploration
- Batch Value-function Approximation with Only Realizability
- Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms
- Reinforcement Learning via Fenchel-Rockafellar Duality
- Is Pessimism Provably Efficient for Offline RL?
- The Importance of Pessimism in Fixed-Dataset Policy Optimization
- Near-Optimal Provable Uniform Convergence in Offline Policy Evaluation for Reinforcement Learning
- Off-Policy Evaluation via the Regularized Lagrangian
- Bilinear Classes: A Structural Framework for Provable Generalization in RL
- Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functions
- Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
- Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RL
- Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparison
- Batch Policy Learning in Average Reward Markov Decision Processes
- Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation
- Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy
- Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning
- Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
- Model-free Representation Learning and Exploration in Low-rank MDPs