Offline RL Without Off-Policy Evaluation
arXiv:2106.08909
Abstract
Most prior approaches to offline reinforcement learning (RL) have taken an iterative actor-critic approach involving off-policy evaluation. In this paper we show that simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well. This one-step algorithm beats the previously reported results of iterative algorithms on a large portion of the D4RL benchmark. The one-step baseline achieves this strong performance while being notably simpler and more robust to hyperparameters than previously proposed iterative algorithms. We argue that the relatively poor performance of iterative approaches is a result of the high variance inherent in doing off-policy evaluation and magnified by the repeated optimization of policies against those estimates. In addition, we hypothesize that the strong performance of the one-step algorithm is due to a combination of favorable structure in the environment and behavior policy.
Thirty-fifth Conference on Neural Information Processing Systems, 2021
References in corpus (12)
- Conservative Q-Learning for Offline Reinforcement Learning
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Benchmarking Batch Deep Reinforcement Learning Algorithms
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
- AlgaeDICE: Policy Gradient from Arbitrary Experience
- Information-Theoretic Considerations in Batch Reinforcement Learning
- Offline Reinforcement Learning with Fisher Divergence Critic Regularization
- A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Setting
- Instabilities of Offline RL with Pre-Trained Neural Representation
- Imitation-Regularized Offline Learning
- Regularized Behavior Value Estimation
- You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL