D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Abstract

The offline reinforcement learning (RL) setting (also known as full batch RL), where a policy is learned from a static dataset, is compelling as progress enables RL methods to take advantage of large, previously-collected datasets, much like how the rise of large datasets has fueled results in supervised learning. However, existing online RL benchmarks are not tailored towards the offline setting and existing offline RL benchmarks are restricted to data generated by partially-trained agents, making progress in offline RL difficult to measure. In this work, we introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. With a focus on dataset collection, examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multitask datasets where an agent performs different tasks in the same environment, and datasets collected with mixtures of policies. By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deficiencies of existing algorithms. To facilitate research, we have released our benchmark tasks and datasets with a comprehensive evaluation of existing algorithms, an evaluation protocol, and open-source examples. This serves as a common starting point for the community to identify shortcomings in existing offline RL methods and a collaborative route for progress in this emerging area.

arXiv commentWebsite available at https://sites.google.com/view/d4rl/home

References in corpus (23)

• Congested Traffic States in Empirical Observations and Microscopic Simulations
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Conservative Q-Learning for Offline Reinforcement Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Challenges of Real-World Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• Learning from Logged Implicit Exploration Data
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Safe Policy Improvement with Baseline Bootstrapping
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Deep Imitative Models for Flexible Inference, Planning, and Control
• RecSim: A Configurable Simulation Platform for Recommender Systems
• An empirical investigation of the challenges of real-world reinforcement learning
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation
• A Real-Time Model-Based Reinforcement Learning Architecture for Robot Control

Cited by in corpus (93)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Offline Reinforcement Learning with Implicit Q-Learning
• Rearrangement: A Challenge for Embodied AI
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• COMBO: Conservative Offline Model-Based Policy Optimization
• Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
• d3rlpy: An Offline Deep Reinforcement Learning Library
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• Is Pessimism Provably Efficient for Offline RL?
• Benchmarks for Deep Off-Policy Evaluation
• Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Review
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Value Penalized Q-Learning for Recommender Systems
• Model-Based Offline Planning
• What Matters for Adversarial Imitation Learning?
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning
• Offline Reinforcement Learning from Images with Latent Space Models
• Offline RL Without Off-Policy Evaluation
• A Maintenance Planning Framework using Online and Offline Deep Reinforcement Learning
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Probabilistic design of optimal sequential decision-making algorithms in learning and control
• Offline Learning from Demonstrations and Unlabeled Experience
• PLAS: Latent Action Space for Offline Reinforcement Learning
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Reset-Free Lifelong Learning with Skill-Space Planning
• Residual Reinforcement Learning from Demonstrations
• Offline Reinforcement Learning with Value-based Episodic Memory
• Multi-task Batch Reinforcement Learning with Metric Learning
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• Demonstration-Guided Reinforcement Learning with Learned Skills
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Offline Reinforcement Learning with Soft Behavior Regularization
• C-Learning: Learning to Achieve Goals via Recursive Classification
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning
• rl_reach: Reproducible Reinforcement Learning Experiments for Robotic Reaching Tasks
• Provable Representation Learning for Imitation with Contrastive Fourier Features
• Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
• Continuous Doubly Constrained Batch Reinforcement Learning
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Offline Reinforcement Learning with Pseudometric Learning
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Interactive Visualization for Debugging RL
• A Pragmatic Look at Deep Imitation Learning
• Regularized Behavior Value Estimation
• HiER: Highlight Experience Replay for Boosting Off-Policy Reinforcement Learning Agents
• Towards Instance-Optimal Offline Reinforcement Learning with Pessimism
• Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
• Offline Reinforcement Learning Hands-On
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning
• Revisiting Prioritized Experience Replay: A Value Perspective
• Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments
• Implicit Behavioral Cloning
• Improving Learning from Demonstrations by Learning from Experience
• Offline Preference-Based Apprenticeship Learning
• You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
• Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
• Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
• Boosting Offline Reinforcement Learning with Residual Generative Modeling
• Learning Control Policies for Variable Objectives from Offline Data
• Configuration Path Control
• Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learning
• Curriculum Offline Imitation Learning
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
• Offline RL With Resource Constrained Online Deployment
• Supervised Off-Policy Ranking
• Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
• Learning Off-Policy with Online Planning
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• TradeR: Practical Deep Hierarchical Reinforcement Learning for Trade Execution
• Offline Inverse Reinforcement Learning
• Quantile Filtered Imitation Learning
• Teachable Reinforcement Learning via Advice Distillation
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning
• Active Offline Policy Selection
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters