Conservative Q-Learning for Offline Reinforcement Learning

Abstract

Effectively leveraging large, previously collected datasets in reinforcement learning (RL) is a key challenge for large-scale real-world applications. Offline RL algorithms promise to learn effective policies from previously-collected, static datasets without further interaction. However, in practice, offline RL presents a major challenge, and standard off-policy RL methods can fail due to overestimation of values induced by the distributional shift between the dataset and the learned policy, especially when training on complex and multi-modal data distributions. In this paper, we propose conservative Q-learning (CQL), which aims to address these limitations by learning a conservative Q-function such that the expected value of a policy under this Q-function lower-bounds its true value. We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees. In practice, CQL augments the standard Bellman error objective with a simple Q-value regularizer which is straightforward to implement on top of existing deep Q-learning and actor-critic implementations. On both discrete and continuous control domains, we show that CQL substantially outperforms existing offline RL methods, often learning policies that attain 2-5 times higher final return, especially when learning from complex and multi-modal data distributions.

Preprint. Website at: https://sites.google.com/view/cql-offline-rl

References in corpus (26)

• Playing Atari with Deep Reinforcement Learning
• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Reinforcement Learning with Deep Energy-Based Policies
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Deep Q-learning from Demonstrations
• Behavior Regularized Offline Reinforcement Learning
• Exploration by Random Network Distillation
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• An Optimistic Perspective on Offline Reinforcement Learning
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Safe Policy Improvement by Minimizing Robust Baseline Regret
• Towards Characterizing Divergence in Deep Q-Learning
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• Learning from Scarce Experience
• Approximate Policy Iteration Schemes: A Comparison
• Consistent On-Line Off-Policy Evaluation
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparison
• Variational Bayesian Reinforcement Learning with Regret Bounds
• A General Family of Robust Stochastic Operators for Reinforcement Learning

Cited by in corpus (117)

• On the Opportunities and Risks of Foundation Models
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• Offline Reinforcement Learning with Implicit Q-Learning
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Real-World Robot Applications of Foundation Models: A Review
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• COMBO: Conservative Offline Model-Based Policy Optimization
• Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems
• Deep reinforcement learning for machine scheduling: Methodology, the state-of-the-art, and future directions
• d3rlpy: An Offline Deep Reinforcement Learning Library
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• Conservative Safety Critics for Exploration
• Learning Robotic Navigation from Experience: Principles, Methods, and Recent Results
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• Is Pessimism Provably Efficient for Offline RL?
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Representation Matters: Offline Pretraining for Sequential Decision Making
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Value Penalized Q-Learning for Recommender Systems
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Evolving Reinforcement Learning Algorithms
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons
• Model-Based Visual Planning with Self-Supervised Functional Distances
• Offline Reinforcement Learning from Images with Latent Space Models
• Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Offline RL Without Off-Policy Evaluation
• Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
• Bellman-consistent Pessimism for Offline Reinforcement Learning
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Probabilistic design of optimal sequential decision-making algorithms in learning and control
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Heuristic-Guided Reinforcement Learning
• Reset-Free Lifelong Learning with Skill-Space Planning
• Multi-task Batch Reinforcement Learning with Metric Learning
• Offline Reinforcement Learning with Value-based Episodic Memory
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Offline Meta Learning of Exploration
• RL-MPCA: A Reinforcement Learning Based Multi-Phase Computation Allocation Approach for Recommender Systems
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation
• Corruption-Robust Offline Reinforcement Learning
• No RL, No Simulation: Learning to Navigate without Navigating
• Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback
• Learning Vision-based Robotic Manipulation Tasks Sequentially in Offline Reinforcement Learning Settings
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Nearly Horizon-Free Offline Reinforcement Learning
• Continuous Doubly Constrained Batch Reinforcement Learning
• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
• Instabilities of Offline RL with Pre-Trained Neural Representation
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
• Visual Adversarial Imitation Learning using Variational Models
• Safe Autonomous Racing via Approximate Reachability on Ego-vision
• A comprehensive approach to incorporating intermolecular dispersion into the openCOSMO-RS model. Part 2: Atomic polarizabilities
• Offline Reinforcement Learning with Pseudometric Learning
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Offline reinforcement learning with uncertainty for treatment strategies in sepsis
• Representation Learning for Online and Offline RL in Low-rank MDPs
• Offline Reinforcement Learning Hands-On
• Regularized Behavior Value Estimation
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
• Multi-task Offline Reinforcement Learning for Online Advertising in Recommender Systems
• Iterative Amortized Policy Optimization
• Safe Driving via Expert Guided Policy Optimization
• BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning
• A view on learning robust goal-conditioned value functions: Interplay between RL and MPC
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Robust Batch Policy Learning in Markov Decision Processes
• On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
• Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learning
• SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Offline Preference-Based Apprenticeship Learning
• Direct Advantage Estimation
• Data-Driven Offline Optimization For Architecting Hardware Accelerators
• Boosting Offline Reinforcement Learning with Residual Generative Modeling
• Offline RL With Resource Constrained Online Deployment
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• Towards robust and domain agnostic reinforcement learning competitions
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• Curriculum Offline Imitation Learning
• UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning
• Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamics
• Learning the Markov Decision Process in the Sparse Gaussian Elimination
• SEIHAI: A Sample-efficient Hierarchical AI for the MineRL Competition
• Learning Off-Policy with Online Planning
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• Data Augmentation through Expert-guided Symmetry Detection to Improve Performance in Offline Reinforcement Learning
• Supervised Off-Policy Ranking
• Active Offline Policy Selection
• Generating GPU Compiler Heuristics using Reinforcement Learning
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
• Fast Block Linear System Solver Using Q-Learning Schduling for Unified Dynamic Power System Simulations
• Nonparametric Additive Value Functions: Interpretable Reinforcement Learning with an Application to Surgical Recovery
• Quantile Filtered Imitation Learning
• OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatment
• Understanding the World Through Action
• Uniformly Conservative Exploration in Reinforcement Learning
• Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems
• Offline Inverse Reinforcement Learning
• Compressive Features in Offline Reinforcement Learning for Recommender Systems
• Enhancing Reinforcement Learning Through Guided Search