A Minimalist Approach to Offline Reinforcement Learning

Abstract

Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data. Due to errors in value estimation from out-of-distribution actions, most offline RL algorithms take the approach of constraining or regularizing the policy with the actions contained in the dataset. Built on pre-existing RL algorithms, modifications to make an RL algorithm work offline comes at the cost of additional complexity. Offline RL algorithms introduce new hyperparameters and often leverage secondary components such as generative models, while adjusting the underlying RL algorithm. In this paper we aim to make a deep RL algorithm work while making minimal changes. We find that we can match the performance of state-of-the-art offline RL algorithms by simply adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data. The resulting algorithm is a simple to implement and tune baseline, while more than halving the overall run time by removing the additional computational overhead of previous methods.

arXiv commentNeurIPS 2021 Spotlight

References in corpus (29)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Continuous control with deep reinforcement learning
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Deep Reinforcement Learning that Matters
• Behavior Regularized Offline Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• Critic Regularized Regression
• Horizon: Facebook's Open Source Applied Reinforcement Learning Platform
• Evaluating Reinforcement Learning Algorithms in Observational Health Settings
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Safe Policy Improvement with Baseline Bootstrapping
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Behaviour Suite for Reinforcement Learning
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Benchmarks for Deep Off-Policy Evaluation
• Offline Policy Selection under Uncertainty

Cited by in corpus (17)

• Offline Reinforcement Learning with Implicit Q-Learning
• d3rlpy: An Offline Deep Reinforcement Learning Library
• Exploration and Regularization of the Latent Action Space in Recommendation
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• Value Penalized Q-Learning for Recommender Systems
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Offline Reinforcement Learning with Soft Behavior Regularization
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Generalized Decision Transformer for Offline Hindsight Information Matching
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning
• Continuous Control with Action Quantization from Demonstrations
• Offline RL With Resource Constrained Online Deployment
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
• Quantile Filtered Imitation Learning