Benchmarking Batch Deep Reinforcement Learning Algorithms

Abstract

Widely-used deep reinforcement learning algorithms have been shown to fail in the batch setting--learning from a fixed data set without interaction with the environment. Following this result, there have been several papers showing reasonable performances under a variety of environments and batch settings. In this paper, we benchmark the performance of recent off-policy and batch reinforcement learning algorithms under unified settings on the Atari domain, with data generated by a single partially-trained behavioral policy. We find that under these conditions, many of these algorithms underperform DQN trained online with the same amount of data, as well as the partially-trained behavioral policy. To introduce a strong baseline, we adapt the Batch-Constrained Q-learning algorithm to a discrete-action setting, and show it outperforms all existing algorithms at this task.

Deep RL Workshop NeurIPS 2019

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Exploration by Random Network Distillation
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Horizon: Facebook's Open Source Applied Reinforcement Learning Platform
• Safe Policy Improvement with Baseline Bootstrapping
• Safe Policy Improvement by Minimizing Robust Baseline Regret

Cited by in corpus (74)

• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• A Theoretical Analysis of Deep Q-Learning
• An Optimistic Perspective on Offline Reinforcement Learning
• Critic Regularized Regression
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• First Order Constrained Optimization in Policy Space
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• d3rlpy: An Offline Deep Reinforcement Learning Library
• Conservative Safety Critics for Exploration
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehicles
• TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learning
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• LTL2Action: Generalizing LTL Instructions for Multi-Task RL
• Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safety
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Risk
• Multi-Agent Constrained Policy Optimisation
• Offline RL Without Off-Policy Evaluation
• Constrained Model-based Reinforcement Learning with Robust Cross-Entropy Method
• Curriculum Learning for Safe Mapless Navigation
• Learning to run a Power Network Challenge: a Retrospective Analysis
• An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare
• PowerGym: A Reinforcement Learning Environment for Volt-Var Control in Power Distribution Systems
• Reset-Free Lifelong Learning with Skill-Space Planning
• An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay
• Why People Skip Music? On Predicting Music Skips using Deep Reinforcement Learning
• Causality and Batch Reinforcement Learning: Complementary Approaches To Planning In Unknown Domains
• Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement
• Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
• ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning
• Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluation
• Safe Reinforcement Learning with Natural Language Constraints
• Balancing Constraints and Rewards with Meta-Gradient D4PG
• Interpretable Model-based Hierarchical Reinforcement Learning using Inductive Logic Programming
• Regularized Behavior Value Estimation
• RL-Scope: Cross-Stack Profiling for Deep Reinforcement Learning Workloads
• Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies
• Offline Reinforcement Learning Hands-On
• GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model
• BCQQ: Batch-Constraint Quantum Q-Learning with Cyclic Data Re-uploading
• Playing Minecraft with Behavioural Cloning
• Formal Language Constraints for Markov Decision Processes
• Approximate Model-Based Shielding for Safe Reinforcement Learning
• TanksWorld: A Multi-Agent Environment for AI Safety Research
• Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments
• Avoiding Side Effects in Complex Environments
• Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
• Exploration by Maximizing Rényi Entropy for Reward-Free RL Framework
• Predicting the Need for Blood Transfusion in Intensive Care Units with Reinforcement Learning
• Challenges for Using Impact Regularizers to Avoid Negative Side Effects
• Learn Zero-Constraint-Violation Policy in Model-Free Constrained Reinforcement Learning
• Interpretable performance analysis towards offline reinforcement learning: A dataset perspective
• Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• Supervised Off-Policy Ranking
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• Concave Utility Reinforcement Learning with Zero-Constraint Violations
• Recurrent Sum-Product-Max Networks for Decision Making in Perfectly-Observed Environments
• Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
• Compressive Features in Offline Reinforcement Learning for Recommender Systems
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
• Learning robust driving policies without online exploration
• Making Human-Like Trade-offs in Constrained Environments by Learning from Demonstrations
• A Policy Efficient Reduction Approach to Convex Constrained Deep Reinforcement Learning
• Batch Reinforcement Learning from Crowds
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms