Benchmarking Batch Deep Reinforcement Learning Algorithms

Abstract

Widely-used deep reinforcement learning algorithms have been shown to fail in the batch setting--learning from a fixed data set without interaction with the environment. Following this result, there have been several papers showing reasonable performances under a variety of environments and batch settings. In this paper, we benchmark the performance of recent off-policy and batch reinforcement learning algorithms under unified settings on the Atari domain, with data generated by a single partially-trained behavioral policy. We find that under these conditions, many of these algorithms underperform DQN trained online with the same amount of data, as well as the partially-trained behavioral policy. To introduce a strong baseline, we adapt the Batch-Constrained Q-learning algorithm to a discrete-action setting, and show it outperforms all existing algorithms at this task.

Deep RL Workshop NeurIPS 2019

References in corpus (3)

• Dopamine: A Research Framework for Deep Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Safe Policy Improvement by Minimizing Robust Baseline Regret

Cited by in corpus (26)

• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safety
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare
• Causality and Batch Reinforcement Learning: Complementary Approaches To Planning In Unknown Domains
• Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement
• Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation
• Interpretable Model-based Hierarchical Reinforcement Learning using Inductive Logic Programming
• Balancing Constraints and Rewards with Meta-Gradient D4PG
• Playing Minecraft with Behavioural Cloning
• Offline Reinforcement Learning Hands-On
• RL-Scope: Cross-Stack Profiling for Deep Reinforcement Learning Workloads
• Regularized Behavior Value Estimation
• TanksWorld: A Multi-Agent Environment for AI Safety Research
• Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
• Challenges for Using Impact Regularizers to Avoid Negative Side Effects
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• Recurrent Sum-Product-Max Networks for Decision Making in Perfectly-Observed Environments
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning
• Making Human-Like Trade-offs in Constrained Environments by Learning from Demonstrations
• Learning robust driving policies without online exploration
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms