Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction

Abstract

Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only limited progress without collecting additional on-policy data. As a step towards more robust off-policy algorithms, we study the setting where the off-policy experience is fixed and there is no further interaction with the environment. We identify bootstrapping error as a key source of instability in current methods. Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator. We theoretically analyze bootstrapping error, and demonstrate how carefully constraining action selection in the backup can mitigate it. Based on our analysis, we propose a practical algorithm, bootstrapping error accumulation reduction (BEAR). We demonstrate that BEAR is able to learn robustly from different off-policy distributions, including random and suboptimal demonstrations, on a range of continuous control tasks.

Accepted at NeurIPS 2019; Project Website: https://sites.google.com/view/bear-off-policyrl

References in corpus (3)

• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment
• Emphatic Temporal-Difference Learning

Cited by in corpus (88)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• A Minimalist Approach to Offline Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• MOReL : Model-Based Offline Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Offline Reinforcement Learning with Implicit Q-Learning
• An Optimistic Perspective on Offline Reinforcement Learning
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• COMBO: Conservative Offline Model-Based Policy Optimization
• An empirical investigation of the challenges of real-world reinforcement learning
• Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• d3rlpy: An Offline Deep Reinforcement Learning Library
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• Learning to Reach Goals via Iterated Supervised Learning
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Value Penalized Q-Learning for Recommender Systems
• CoinDICE: Off-Policy Confidence Interval Estimation
• Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping
• Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding
• On the model-based stochastic value gradient for continuous reinforcement learning
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Offline RL Without Off-Policy Evaluation
• Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning
• Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
• Bellman-consistent Pessimism for Offline Reinforcement Learning
• Efficient (Soft) Q-Learning for Text Generation with Limited Good Data
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• DraftRec: Personalized Draft Recommendation for Winning in Multi-Player Online Battle Arena Games
• Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations
• Student-Initiated Action Advising via Advice Novelty
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Multi-task Batch Reinforcement Learning with Metric Learning
• Robotic Surgery With Lean Reinforcement Learning
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Why People Skip Music? On Predicting Music Skips using Deep Reinforcement Learning
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Medical Dead-ends and Learning to Identify High-risk States and Treatments
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• Corruption-Robust Offline Reinforcement Learning
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
• Continuous Doubly Constrained Batch Reinforcement Learning
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Offline reinforcement learning with uncertainty for treatment strategies in sepsis
• TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
• Offline Reinforcement Learning with Pseudometric Learning
• Iterative Amortized Policy Optimization
• AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
• Offline Decentralized Multi-Agent Reinforcement Learning
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• Consolidated Adaptive T-soft Update for Deep Reinforcement Learning
• Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
• Learning Control Policies for Variable Objectives from Offline Data
• Offline Preference-Based Apprenticeship Learning
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
• Online Safety Assurance for Deep Reinforcement Learning
• Zeroth-Order Supervised Policy Improvement
• SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations
• Supervised Off-Policy Ranking
• Offline RL With Resource Constrained Online Deployment
• Interpretable performance analysis towards offline reinforcement learning: A dataset perspective
• SEIHAI: A Sample-efficient Hierarchical AI for the MineRL Competition
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
• Learning Off-Policy with Online Planning
• Curriculum Offline Imitation Learning
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• Generative Temporal Difference Learning for Infinite-Horizon Prediction
• Batch Reinforcement Learning from Crowds
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• Time-Aware Q-Networks: Resolving Temporal Irregularity for Deep Reinforcement Learning
• Offline Inverse Reinforcement Learning
• Variance-Aware Off-Policy Evaluation with Linear Function Approximation
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control
• Understanding the World Through Action