Boosting Offline Reinforcement Learning with Residual Generative Modeling
arXiv:2106.10411
Abstract
Offline reinforcement learning (RL) tries to learn the near-optimal policy with recorded offline experience without online exploration. Current offline RL research includes: 1) generative modeling, i.e., approximating a policy using fixed data; and 2) learning the state-action value function. While most research focuses on the state-action function part through reducing the bootstrapping error in value function approximation induced by the distribution shift of training data, the effects of error propagation in generative modeling have been neglected. In this paper, we analyze the error in generative modeling. We propose AQL (action-conditioned Q-learning), a residual generative model to reduce policy approximation error for offline RL. We show that our method can learn more accurate policy approximations in different benchmark datasets. In addition, we show that the proposed offline RL method can learn more competitive AI agents in complex control tasks under the multiplayer online battle arena (MOBA) game Honor of Kings.
arXiv commentAccepted by IJCAI 2021, appendix included, 9 pages, 4 figures, 2 tables
References in corpus (15)
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Dota 2 with Large Scale Deep Reinforcement Learning
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- Conservative Q-Learning for Offline Reinforcement Learning
- Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Challenges of Real-World Reinforcement Learning
- Behavior Regularized Offline Reinforcement Learning
- Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings
- Are ResNets Provably Better than Linear Predictors?
- Learning Deep ResNet Blocks Sequentially using Boosting Theory
- Which Heroes to Pick? Learning to Draft in MOBA Games with Neural Networks and Tree Search