Critic Regularized Regression

Abstract

Offline reinforcement learning (RL), also known as batch RL, offers the prospect of policy optimization from large pre-recorded datasets without online environment interaction. It addresses challenges with regard to the cost of data collection and safety, both of which are particularly pertinent to real-world applications of RL. Unfortunately, most off-policy algorithms perform poorly when learning from a fixed dataset. In this paper, we propose a novel offline RL algorithm to learn policies from data using a form of critic-regularized regression (CRR). We find that CRR performs surprisingly well and scales to tasks with high-dimensional state and action spaces -- outperforming several state-of-the-art offline RL algorithms by a significant margin on a wide range of benchmark tasks.

arXiv comment24 pages; presented at NeurIPS 2020

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Emergence of Locomotion Behaviours in Rich Environments
• DeepMind Control Suite
• Learning Continuous Control Policies by Stochastic Value Gradients
• Challenges of Real-World Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Neural probabilistic motor primitives for humanoid control
• Hierarchical visuomotor control of humanoids
• Acme: A Research Framework for Distributed Reinforcement Learning
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• Reward-Conditioned Policies

Cited by in corpus (40)

• A Minimalist Approach to Offline Reinforcement Learning
• Offline Reinforcement Learning with Implicit Q-Learning
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• An empirical investigation of the challenges of real-world reinforcement learning
• d3rlpy: An Offline Deep Reinforcement Learning Library
• Learning Robotic Navigation from Experience: Principles, Methods, and Recent Results
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• Is Pessimism Provably Efficient for Offline RL?
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Benchmarks for Deep Off-Policy Evaluation
• Model-Based Offline Planning
• Offline RL Without Off-Policy Evaluation
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Offline Learning from Demonstrations and Unlabeled Experience
• Offline Reinforcement Learning with Soft Behavior Regularization
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• Medical Dead-ends and Learning to Identify High-risk States and Treatments
• Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation
• Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning
• Continuous Doubly Constrained Batch Reinforcement Learning
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Offline Reinforcement Learning Hands-On
• Regularized Behavior Value Estimation
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• REPAINT: Knowledge Transfer in Deep Reinforcement Learning
• PACE: Data-Driven Virtual Agent Interaction in Dense and Cluttered Environments
• You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
• Learning Off-Policy with Online Planning
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• Reinforcement Learning for Vision-based Object Manipulation with Non-parametric Policy and Action Primitives
• Supervised Off-Policy Ranking
• Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
• Active Offline Policy Selection