Behavior Regularized Offline Reinforcement Learning

Abstract

In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline dataset of logged experience. In such settings, standard RL algorithms have been shown to diverge or otherwise yield poor performance. Accordingly, recent work has suggested a number of remedies to these issues. In this work, we introduce a general framework, behavior regularized actor critic (BRAC), to empirically evaluate recently proposed methods as well as a number of simple baselines across a variety of offline continuous control tasks. Surprisingly, we find that many of the technical complexities introduced in recent methods are unnecessary to achieve strong performance. Additional ablations provide insights into which design choices matter most in the offline RL setting.

References in corpus (12)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• Maximum a Posteriori Policy Optimisation
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Deep Reinforcement Learning and the Deadly Triad
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Diagnosing Bottlenecks in Deep Q-learning Algorithms
• Off-Policy Evaluation via Off-Policy Classification

Cited by in corpus (90)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• A Minimalist Approach to Offline Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• Offline Reinforcement Learning with Implicit Q-Learning
• An Optimistic Perspective on Offline Reinforcement Learning
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• COMBO: Conservative Offline Model-Based Policy Optimization
• An empirical investigation of the challenges of real-world reinforcement learning
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Value Penalized Q-Learning for Recommender Systems
• CoinDICE: Off-Policy Confidence Interval Estimation
• Model-Based Offline Planning
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Model-Based Visual Planning with Self-Supervised Functional Distances
• On the model-based stochastic value gradient for continuous reinforcement learning
• Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• PLAS: Latent Action Space for Offline Reinforcement Learning
• DraftRec: Personalized Draft Recommendation for Winning in Multi-Player Online Battle Arena Games
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Reset-Free Lifelong Learning with Skill-Space Planning
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Offline Reinforcement Learning with Soft Behavior Regularization
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• Medical Dead-ends and Learning to Identify High-risk States and Treatments
• Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
• Corruption-Robust Offline Reinforcement Learning
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• Generalized Policy Improvement Algorithms with Theoretically Supported Sample Reuse
• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Continuous Doubly Constrained Batch Reinforcement Learning
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Offline Reinforcement Learning with Pseudometric Learning
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
• Safe Driving via Expert Guided Policy Optimization
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Learning Control Policies for Variable Objectives from Offline Data
• Improving Learning from Demonstrations by Learning from Experience
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
• You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
• SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations
• Boosting Offline Reinforcement Learning with Residual Generative Modeling
• Continuous Control with Action Quantization from Demonstrations
• Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• Offline RL With Resource Constrained Online Deployment
• Regularized Policies are Reward Robust
• Dual Behavior Regularized Reinforcement Learning
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learning
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• Reinforcement Learning for Vision-based Object Manipulation with Non-parametric Policy and Action Primitives
• Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularization
• Enhancing Reinforcement Learning Through Guided Search
• Offline Inverse Reinforcement Learning
• Off-Policy Self-Critical Training for Transformer in Visual Paragraph Generation
• Learning Personalized Discretionary Lane-Change Initiation for Fully Autonomous Driving Based on Reinforcement Learning
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning