Offline Reinforcement Learning with Soft Behavior Regularization

Abstract

Most prior approaches to offline reinforcement learning (RL) utilize \textit{behavior regularization}, typically augmenting existing off-policy actor critic algorithms with a penalty measuring divergence between the policy and the offline data. However, these approaches lack guaranteed performance improvement over the behavior policy. In this work, we start from the performance difference between the learned policy and the behavior policy, we derive a new policy learning objective that can be used in the offline setting, which corresponds to the advantage function value of the behavior policy, multiplying by a state-marginal density ratio. We propose a practical way to compute the density ratio and demonstrate its equivalence to a state-dependent behavior regularization. Unlike state-independent regularization used in prior approaches, this \textit{soft} regularization allows more freedom of policy deviation at high confidence states, leading to better performance and stability. We thus term our resulting algorithm Soft Behavior-regularized Actor Critic (SBAC). Our experimental results show that SBAC matches or outperforms the state-of-the-art on a set of continuous control locomotion and manipulation tasks.

References in corpus (19)

• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• Critic Regularized Regression
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Safe Policy Improvement with Baseline Bootstrapping
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Reinforcement Learning based Recommender System using Biclustering Technique
• Off-Policy Actor-Critic
• Truly Proximal Policy Optimization
• Constrained Markov Decision Processes via Backward Value Functions
• Offline RL Without Off-Policy Evaluation
• Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning