Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Abstract

In this tutorial article, we aim to provide the reader with the conceptual tools needed to get started on research on offline reinforcement learning algorithms: reinforcement learning algorithms that utilize previously collected data, without additional online data collection. Offline reinforcement learning algorithms hold tremendous promise for making it possible to turn large datasets into powerful decision making engines. Effective offline reinforcement learning methods would be able to extract policies with the maximum possible utility out of the available data, thereby allowing automation of a wide range of decision-making domains, from healthcare and education to robotics. However, the limitations of current algorithms make this difficult. We will aim to provide the reader with an understanding of these challenges, particularly in the context of modern deep reinforcement learning methods, and describe some potential solutions that have been explored in recent work to mitigate these challenges, along with recent applications, and a discussion of perspectives on open problems in the field.

References in corpus (73)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• End to End Learning for Self-Driving Cars
• Trust Region Policy Optimization
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• A Contextual-Bandit Approach to Personalized News Article Recommendation
• A Survey of Autonomous Driving: Common Practices and Emerging Technologies
• Semi-Supervised Learning with Deep Generative Models
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Interaction Networks for Learning about Objects, Relations and Physics
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Conservative Q-Learning for Offline Reinforcement Learning
• Model-Based Reinforcement Learning for Atari
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Learning Continuous Control Policies by Stochastic Value Gradients
• Certifying Some Distributional Robustness with Principled Adversarial Training
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Invariant Risk Minimization
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Sample Efficient Actor-Critic with Experience Replay
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Causality for Machine Learning
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Learning from Logged Implicit Exploration Data
• Deep Reinforcement Learning and the Deadly Triad
• On integral probability metrics, ϕ-divergences and binary classification
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Evaluating Reinforcement Learning Algorithms in Observational Health Settings
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Safe Policy Improvement by Minimizing Robust Baseline Regret
• Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment
• Deep Imitative Models for Flexible Inference, Planning, and Control
• On Convergence of Emphatic Temporal-Difference Learning
• GenDICE: Generalized Offline Estimation of Stationary Values
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• Learning from Scarce Experience
• Deep Reinforcement Learning for Sepsis Treatment
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Off-Policy Policy Gradient with State Distribution Correction
• Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning
• Consistent On-Line Off-Policy Evaluation
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient
• Minimax Weight and Q-Function Learning for Off-Policy Evaluation
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
• The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation
• Learning from Conditional Distributions via Dual Embeddings
• Dual Policy Iteration
• Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis
• RoboNet: Large-Scale Multi-Robot Learning
• Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation
• Batch Stationary Distribution Estimation
• Stochastic Primal-Dual Q-Learning
• Learning When-to-Treat Policies
• Boosting the Actor with Dual Critic
• From Importance Sampling to Doubly Robust Policy Gradient
• Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods
• Reward-estimation variance elimination in sequential decision processes
• Emphatic TD Bellman Operator is a Contraction

Cited by in corpus (216)

• Conservative Q-Learning for Offline Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges
• Advances and Challenges in Conversational Recommender Systems: A Survey
• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• Data-driven Machinery Fault Diagnosis: A Comprehensive Review
• MOReL : Model-Based Offline Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning
• Review: Deep Learning in Electron Microscopy
• Critic Regularized Regression
• Data-Efficient Deep Reinforcement Learning for Attitude Control of Fixed-Wing UAVs: Field Experiments
• Machine learning for industrial sensing and control: A survey and practical perspective
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• COMBO: Conservative Offline Model-Based Policy Optimization
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• An empirical investigation of the challenges of real-world reinforcement learning
• Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
• Modified DDPG car-following model with a real-world human driving experience with CARLA simulator
• d3rlpy: An Offline Deep Reinforcement Learning Library
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model
• From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
• Conservative Safety Critics for Exploration
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• Bridging adaptive management and reinforcement learning for more robust decisions
• Action valuation of on- and off-ball soccer players based on multi-agent deep reinforcement learning
• Learning Robotic Navigation from Experience: Principles, Methods, and Recent Results
• ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• Is Pessimism Provably Efficient for Offline RL?
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Causal Reinforcement Learning using Observational and Interventional Data
• Rethinking Closed-loop Training for Autonomous Driving
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation
• Value Penalized Q-Learning for Recommender Systems
• A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions
• Model-Based Offline Planning
• Training Agents using Upside-Down Reinforcement Learning
• Near-Optimal Offline Reinforcement Learning via Double Variance Reduction
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Model-Based Visual Planning with Self-Supervised Functional Distances
• Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others
• Offline Reinforcement Learning from Images with Latent Space Models
• Autofocused oracles for model-based design
• Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning
• Offline RL Without Off-Policy Evaluation
• Bellman-consistent Pessimism for Offline Reinforcement Learning
• Generation and storage of spin squeezing via learning-assisted optimal control
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Probabilistic design of optimal sequential decision-making algorithms in learning and control
• Offline Learning from Demonstrations and Unlabeled Experience
• Offline Supervised Learning V.S. Online Direct Policy Optimization: A Comparative Study and A Unified Training Paradigm for Neural Network-Based Optimal Feedback Control
• Off-Policy Reinforcement Learning with Delayed Rewards
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• PLAS: Latent Action Space for Offline Reinforcement Learning
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data
• Muesli: Combining Improvements in Policy Optimization
• Network Support for High-performance Distributed Machine Learning
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Multi-level Traffic-Responsive Tilt Camera Surveillance through Predictive Correlated Online Learning
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving
• Reset-Free Lifelong Learning with Skill-Space Planning
• On component interactions in two-stage recommender systems
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• Latent feedback control of distributed systems in multiple scenarios through deep learning-based reduced order models
• Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient
• Offline Reinforcement Learning with Value-based Episodic Memory
• Real-time optimal control of high-dimensional parametrized systems by deep learning-based reduced order models
• Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement
• Offline Meta Learning of Exploration
• Learning predictive representations in autonomous driving to improve deep reinforcement learning
• Offline Reinforcement Learning with Soft Behavior Regularization
• Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization
• Why People Skip Music? On Predicting Music Skips using Deep Reinforcement Learning
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation
• Predicting Infectiousness for Proactive Contact Tracing
• Self-Imitation Advantage Learning
• ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning
• Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation
• Ad-load Balancing via Off-policy Learning in a Content Marketplace
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• A New Autoregressive Neural Network Model with Command Compensation for Imitation Learning Based on Bilateral Control
• Medical Dead-ends and Learning to Identify High-risk States and Treatments
• Corruption-Robust Offline Reinforcement Learning
• Provable Representation Learning for Imitation with Contrastive Fourier Features
• Performance guaranteed MPC Policy Approximation via Cost Guided Learning
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithm
• No RL, No Simulation: Learning to Navigate without Navigating
• Semi-supervised reward learning for offline reinforcement learning
• SIRL: Similarity-based Implicit Representation Learning
• Umbrella Reinforcement Learning -- computationally efficient tool for hard non-linear problems
• Instabilities of Offline RL with Pre-Trained Neural Representation
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
• Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
• Nearly Horizon-Free Offline Reinforcement Learning
• Continuous Doubly Constrained Batch Reinforcement Learning
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• Offline reinforcement learning with uncertainty for treatment strategies in sepsis
• Action Guidance: Getting the Best of Sparse Rewards and Shaped Rewards for Real-time Strategy Games
• Reimagining an autonomous vehicle
• Offline Reinforcement Learning with Pseudometric Learning
• Text Generation by Learning from Demonstrations
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Generalized Decision Transformer for Offline Hindsight Information Matching
• A comprehensive approach to incorporating intermolecular dispersion into the openCOSMO-RS model. Part 2: Atomic polarizabilities
• Offline Reinforcement Learning Hands-On
• Regularized Behavior Value Estimation
• Towards Instance-Optimal Offline Reinforcement Learning with Pessimism
• Infinite-Horizon Offline Reinforcement Learning with Linear Function Approximation: Curse of Dimensionality and Algorithm
• C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks
• BCQQ: Batch-Constraint Quantum Q-Learning with Cyclic Data Re-uploading
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning
• Mixture of Step Returns in Bootstrapped DQN
• REALab: An Embedded Perspective on Tampering
• On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning
• BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Safe Policy Improvement Approaches on Discrete Markov Decision Processes
• Offline Decentralized Multi-Agent Reinforcement Learning
• On Instrumental Variable Regression for Deep Offline Policy Evaluation
• Robust Batch Policy Learning in Markov Decision Processes
• Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds
• Towards Robust Off-Policy Evaluation via Human Inputs
• Soft Sensors and Process Control using AI and Dynamic Simulation
• Expert-Guided Symmetry Detection in Markov Decision Processes
• You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
• Causal Multi-Agent Reinforcement Learning: Review and Open Problems
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Learning Control Policies for Variable Objectives from Offline Data
• SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations
• Towards Content Provider Aware Recommender Systems: A Simulation Study on the Interplay between User and Provider Utilities
• Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
• Offline Preference-Based Apprenticeship Learning
• Direct Advantage Estimation
• Online Safety Assurance for Deep Reinforcement Learning
• Self-Imitation Learning by Planning
• Boosting Offline Reinforcement Learning with Residual Generative Modeling
• Offline Neural Contextual Bandits: Pessimism, Optimization and Generalization
• On the Optimality of Batch Policy Optimization Algorithms
• Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective
• Data Augmentation through Expert-guided Symmetry Detection to Improve Performance in Offline Reinforcement Learning
• On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction
• Offline RL With Resource Constrained Online Deployment
• Reinforcement Learning as Iterative and Amortised Inference
• Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee
• Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
• UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning
• Sample Complexity of Offline Reinforcement Learning with Deep ReLU Networks
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• Affine Transport for Sim-to-Real Domain Adaptation
• General Characterization of Agents by States they Visit
• Combining Online Learning and Offline Learning for Contextual Bandits with Deficient Support
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• Learning the Markov Decision Process in the Sparse Gaussian Elimination
• Learning Off-Policy with Online Planning
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Supervised Off-Policy Ranking
• Offline Inverse Reinforcement Learning
• Offline Contextual Bandits for Wireless Network Optimization
• False Correlation Reduction for Offline Reinforcement Learning
• Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings
• Uniformly Conservative Exploration in Reinforcement Learning
• Solving the Real Robot Challenge using Deep Reinforcement Learning
• Risk Sensitive Path Integral Control for Infinite Horizon Problem Formulations
• Video2Skill: Adapting Events in Demonstration Videos to Skills in an Environment using Cyclic MDP Homomorphisms
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning
• Time-Aware Q-Networks: Resolving Temporal Irregularity for Deep Reinforcement Learning
• Regularize! Don't Mix: Multi-Agent Reinforcement Learning without Explicit Centralized Structures
• Mitigation of Adversarial Policy Imitation via Constrained Randomization of Policy (CRoP)
• Learning robust driving policies without online exploration
• Utilizing Skipped Frames in Action Repeats via Pseudo-Actions
• Variance-Aware Off-Policy Evaluation with Linear Function Approximation
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
• Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning
• Fast Block Linear System Solver Using Q-Learning Schduling for Unified Dynamic Power System Simulations
• Understanding the World Through Action
• Learning Robust Controllers Via Probabilistic Model-Based Policy Search
• Active Offline Policy Selection
• Enhancing Reinforcement Learning Through Guided Search
• Remember what you did so you know what to do next
• A Policy Efficient Reduction Approach to Convex Constrained Deep Reinforcement Learning
• Compressive Features in Offline Reinforcement Learning for Recommender Systems
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• Quantile Filtered Imitation Learning
• Learning by Doing: Controlling a Dynamical System using Causality, Control, and Reinforcement Learning
• Transfer learning with causal counterfactual reasoning in Decision Transformers
• Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning