BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning

Abstract

There has recently been a surge in research in batch Deep Reinforcement Learning (DRL), which aims for learning a high-performing policy from a given dataset without additional interactions with the environment. We propose a new algorithm, Best-Action Imitation Learning (BAIL), which strives for both simplicity and performance. BAIL learns a V function, uses the V function to select actions it believes to be high-performing, and then uses those actions to train a policy network using imitation learning. For the MuJoCo benchmark, we provide a comprehensive experimental study of BAIL, comparing its performance to four other batch Q-learning and imitation-learning schemes for a large variety of batch datasets. Our experiments show that BAIL's performance is much higher than the other schemes, and is also computationally much faster than the batch Q-learning schemes.

27 pages(15 pages for appendix); Published in 34th Conference on Neural Information Processing Systems (NeurIPS 2020)

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Deep Q-learning from Demonstrations
• Behavior Regularized Offline Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Learning from Logged Implicit Exploration Data
• Model-Free Imitation Learning with Policy Optimization
• Off-Policy Evaluation via Off-Policy Classification

Cited by in corpus (7)

• Critic Regularized Regression
• An empirical investigation of the challenges of real-world reinforcement learning
• Multi-task Batch Reinforcement Learning with Metric Learning
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Semi-supervised reward learning for offline reinforcement learning
• Interpretable performance analysis towards offline reinforcement learning: A dataset perspective
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets