Behavior Regularized Offline Reinforcement Learning

Abstract

In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline dataset of logged experience. In such settings, standard RL algorithms have been shown to diverge or otherwise yield poor performance. Accordingly, recent work has suggested a number of remedies to these issues. In this work, we introduce a general framework, behavior regularized actor critic (BRAC), to empirically evaluate recently proposed methods as well as a number of simple baselines across a variety of offline continuous control tasks. Surprisingly, we find that many of the technical complexities introduced in recent methods are unnecessary to achieve strong performance. Additional ablations provide insights into which design choices matter most in the offline RL setting.

References in corpus (2)

• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Diagnosing Bottlenecks in Deep Q-learning Algorithms

Cited by in corpus (41)

• Conservative Q-Learning for Offline Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Offline Reinforcement Learning with Implicit Q-Learning
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Model-Based Visual Planning with Self-Supervised Functional Distances
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• PLAS: Latent Action Space for Offline Reinforcement Learning
• DraftRec: Personalized Draft Recommendation for Winning in Multi-Player Online Battle Arena Games
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• Offline Reinforcement Learning with Soft Behavior Regularization
• Corruption-Robust Offline Reinforcement Learning
• Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
• Safe Driving via Expert Guided Policy Optimization
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations
• You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
• Boosting Offline Reinforcement Learning with Residual Generative Modeling
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• Regularized Policies are Reward Robust
• Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularization
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• Dual Behavior Regularized Reinforcement Learning
• Learning Personalized Discretionary Lane-Change Initiation for Fully Autonomous Driving Based on Reinforcement Learning
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning
• Offline Inverse Reinforcement Learning
• Off-Policy Self-Critical Training for Transformer in Visual Paragraph Generation