Regularized Behavior Value Estimation

Abstract

Offline reinforcement learning restricts the learning process to rely only on logged-data without access to an environment. While this enables real-world applications, it also poses unique challenges. One important challenge is dealing with errors caused by the overestimation of values for state-action pairs not well-covered by the training data. Due to bootstrapping, these errors get amplified during training and can lead to divergence, thereby crippling learning. To overcome this challenge, we introduce Regularized Behavior Value Estimation (R-BVE). Unlike most approaches, which use policy improvement during training, R-BVE estimates the value of the behavior policy during training and only performs policy improvement at deployment time. Further, R-BVE uses a ranking regularisation term that favours actions in the dataset that lead to successful outcomes. We provide ample empirical evidence of R-BVE's effectiveness, including state-of-the-art performance on the RL Unplugged ATARI dataset. We also test R-BVE on new datasets, from bsuite and a challenging DeepMind Lab task, and show that R-BVE outperforms other state-of-the-art discrete control offline RL methods.

References in corpus (18)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Dota 2 with Large Scale Deep Reinforcement Learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• MOReL : Model-Based Offline Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• An Optimistic Perspective on Offline Reinforcement Learning
• Critic Regularized Regression
• Observe and Look Further: Achieving Consistent Performance on Atari
• Acme: A Research Framework for Distributed Reinforcement Learning
• Towards Characterizing Divergence in Deep Q-Learning
• Behaviour Suite for Reinforcement Learning
• ConQUR: Mitigating Delusional Bias in Deep Q-learning

Cited by in corpus (4)

• Offline RL Without Off-Policy Evaluation
• Heuristic-Guided Reinforcement Learning
• Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
• Active Offline Policy Selection