Offline RL Without Off-Policy Evaluation
arXiv:2106.08909
Abstract
Most prior approaches to offline reinforcement learning (RL) have taken an iterative actor-critic approach involving off-policy evaluation. In this paper we show that simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well. This one-step algorithm beats the previously reported results of iterative algorithms on a large portion of the D4RL benchmark. The one-step baseline achieves this strong performance while being notably simpler and more robust to hyperparameters than previously proposed iterative algorithms. We argue that the relatively poor performance of iterative approaches is a result of the high variance inherent in doing off-policy evaluation and magnified by the repeated optimization of policies against those estimates. In addition, we hypothesize that the strong performance of the one-step algorithm is due to a combination of favorable structure in the environment and behavior policy.
arXiv commentThirty-fifth Conference on Neural Information Processing Systems, 2021
References in corpus (23)
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Conservative Q-Learning for Offline Reinforcement Learning
- Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Benchmarking Batch Deep Reinforcement Learning Algorithms
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
- Critic Regularized Regression
- AlgaeDICE: Policy Gradient from Arbitrary Experience
- Safe Policy Improvement with Baseline Bootstrapping
- Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
- Information-Theoretic Considerations in Batch Reinforcement Learning
- What are the Statistical Limits of Offline RL with Linear Function Approximation?
- Offline Reinforcement Learning with Fisher Divergence Critic Regularization
- The Importance of Pessimism in Fixed-Dataset Policy Optimization
- Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RL
- A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Setting
- Instabilities of Offline RL with Pre-Trained Neural Representation
- Imitation-Regularized Offline Learning
- Regularized Behavior Value Estimation
- You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL