Offline Reinforcement Learning with Fisher Divergence Critic Regularization

Abstract

Many modern approaches to offline Reinforcement Learning (RL) utilize behavior regularization, typically augmenting a model-free actor critic algorithm with a penalty measuring divergence of the policy from the offline data. In this work, we propose an alternative approach to encouraging the learned policy to stay close to the data, namely parameterizing the critic as the log-behavior-policy, which generated the offline data, plus a state-action value offset term, which can be learned using a neural network. Behavior regularization then corresponds to an appropriate regularizer on the offset term. We propose using a gradient penalty regularizer for the offset term and demonstrate its equivalence to Fisher divergence regularization, suggesting connections to the score matching and generative energy-based model literature. We thus term our resulting algorithm Fisher-BRC (Behavior Regularized Critic). On standard offline RL benchmarks, Fisher-BRC achieves both improved performance and faster convergence over existing state-of-the-art methods.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Soft Actor-Critic Algorithms and Applications
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Reinforcement Learning with Deep Energy-Based Policies
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Optimal control as a graphical model inference problem
• Behavior Regularized Offline Reinforcement Learning
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Taming the Noise in Reinforcement Learning via Soft Updates
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Model-Based Offline Planning
• Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models

Cited by in corpus (7)

• A Minimalist Approach to Offline Reinforcement Learning
• Value Penalized Q-Learning for Recommender Systems
• Offline RL Without Off-Policy Evaluation
• Fast approximations of the Jeffreys divergence between univariate Gaussian mixture models via exponential polynomial densities
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Offline RL With Resource Constrained Online Deployment
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters