Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
arXiv:2105.08140
Abstract
Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. However, existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from out-of-distribution (OOD) actions or states. We hypothesize that a key missing ingredient from the existing methods is a proper treatment of uncertainty in the offline setting. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that detects OOD state-action pairs and down-weights their contribution in the training objectives accordingly. Implementation-wise, we adopt a practical and effective dropout-based uncertainty estimation method that introduces very little overhead over existing RL algorithms. Empirically, we observe that UWAC substantially improves model stability during training. In addition, UWAC out-performs existing offline RL methods on a variety of competitive tasks, and achieves significant performance gains over the state-of-the-art baseline on datasets with sparse demonstrations collected from human experts.
arXiv commentTo appear in ICML 2021
References in corpus (16)
- Playing Atari with Deep Reinforcement Learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Conservative Q-Learning for Offline Reinforcement Learning
- Variational Dropout and the Local Reparameterization Trick
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Behavior Regularized Offline Reinforcement Learning
- MOPO: Model-based Offline Policy Optimization
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
- Distributional Reinforcement Learning with Quantile Regression
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
- BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems
- Safe Policy Improvement with Baseline Bootstrapping
- AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
- Estimating Risk and Uncertainty in Deep Reinforcement Learning
- Variational Deep Q Network