D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Abstract

The offline reinforcement learning (RL) setting (also known as full batch RL), where a policy is learned from a static dataset, is compelling as progress enables RL methods to take advantage of large, previously-collected datasets, much like how the rise of large datasets has fueled results in supervised learning. However, existing online RL benchmarks are not tailored towards the offline setting and existing offline RL benchmarks are restricted to data generated by partially-trained agents, making progress in offline RL difficult to measure. In this work, we introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. With a focus on dataset collection, examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multitask datasets where an agent performs different tasks in the same environment, and datasets collected with mixtures of policies. By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deficiencies of existing algorithms. To facilitate research, we have released our benchmark tasks and datasets with a comprehensive evaluation of existing algorithms, an evaluation protocol, and open-source examples. This serves as a common starting point for the community to identify shortcomings in existing offline RL methods and a collaborative route for progress in this emerging area.

Website available at https://sites.google.com/view/d4rl/home

References in corpus (12)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Challenges of Real-World Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Learning from Logged Implicit Exploration Data
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• RecSim: A Configurable Simulation Platform for Recommender Systems
• An empirical investigation of the challenges of real-world reinforcement learning

Cited by in corpus (91)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Offline Reinforcement Learning with Implicit Q-Learning
• Rearrangement: A Challenge for Embodied AI
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• COMBO: Conservative Offline Model-Based Policy Optimization
• Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
• d3rlpy: An Offline Deep Reinforcement Learning Library
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• Is Pessimism Provably Efficient for Offline RL?
• Benchmarks for Deep Off-Policy Evaluation
• Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Review
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
• Value Penalized Q-Learning for Recommender Systems
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Model-Based Offline Planning
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• What Matters for Adversarial Imitation Learning?
• Offline Reinforcement Learning from Images with Latent Space Models
• Offline RL Without Off-Policy Evaluation
• Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning
• A Maintenance Planning Framework using Online and Offline Deep Reinforcement Learning
• Probabilistic design of optimal sequential decision-making algorithms in learning and control
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Offline Learning from Demonstrations and Unlabeled Experience
• PLAS: Latent Action Space for Offline Reinforcement Learning
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Residual Reinforcement Learning from Demonstrations
• Reset-Free Lifelong Learning with Skill-Space Planning
• Offline Reinforcement Learning with Value-based Episodic Memory
• Demonstration-Guided Reinforcement Learning with Learned Skills
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• Offline Reinforcement Learning with Soft Behavior Regularization
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• C-Learning: Learning to Achieve Goals via Recursive Classification
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning
• rl_reach: Reproducible Reinforcement Learning Experiments for Robotic Reaching Tasks
• Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
• Provable Representation Learning for Imitation with Contrastive Fourier Features
• Continuous Doubly Constrained Batch Reinforcement Learning
• Offline Reinforcement Learning with Pseudometric Learning
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Generalized Decision Transformer for Offline Hindsight Information Matching
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Interactive Visualization for Debugging RL
• Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
• A Pragmatic Look at Deep Imitation Learning
• Offline Reinforcement Learning Hands-On
• Towards Instance-Optimal Offline Reinforcement Learning with Pessimism
• Regularized Behavior Value Estimation
• HiER: Highlight Experience Replay for Boosting Off-Policy Reinforcement Learning Agents
• Implicit Behavioral Cloning
• Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• Revisiting Prioritized Experience Replay: A Value Perspective
• BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning
• Learning Control Policies for Variable Objectives from Offline Data
• Improving Learning from Demonstrations by Learning from Experience
• Boosting Offline Reinforcement Learning with Residual Generative Modeling
• Offline Preference-Based Apprenticeship Learning
• Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
• Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
• You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
• Configuration Path Control
• Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
• Learning Off-Policy with Online Planning
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Supervised Off-Policy Ranking
• Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learning
• Offline RL With Resource Constrained Online Deployment
• Curriculum Offline Imitation Learning
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• Teachable Reinforcement Learning via Advice Distillation
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
• TradeR: Practical Deep Hierarchical Reinforcement Learning for Trade Execution
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning
• Offline Inverse Reinforcement Learning
• Quantile Filtered Imitation Learning
• Active Offline Policy Selection