AWAC: Accelerating Online Reinforcement Learning with Offline Datasets

Abstract

Reinforcement learning (RL) provides an appealing formalism for learning control policies from experience. However, the classic active formulation of RL necessitates a lengthy active exploration process for each behavior, making it difficult to apply in real-world settings such as robotic control. If we can instead allow RL algorithms to effectively use previously collected data to aid the online learning process, such applications could be made substantially more practical: the prior data would provide a starting point that mitigates challenges due to exploration and sample complexity, while the online training enables the agent to perfect the desired skill. Such prior data could either constitute expert demonstrations or sub-optimal prior data that illustrates potentially useful transitions. While a number of prior methods have either used optimal demonstrations to bootstrap RL, or have used sub-optimal data to train purely offline, it remains exceptionally difficult to train a policy with offline data and actually continue to improve it further with online RL. In this paper we analyze why this problem is so challenging, and propose an algorithm that combines sample efficient dynamic programming with maximum likelihood policy updates, providing a simple and effective framework that is able to leverage large amounts of offline data and then quickly perform online fine-tuning of RL policies. We show that our method, advantage weighted actor critic (AWAC), enables rapid learning of skills with a combination of prior demonstration data and online experience. We demonstrate these benefits on simulated and real-world robotics domains, including dexterous manipulation with a real multi-fingered hand, drawer opening with a robotic arm, and rotating a valve. Our results show that incorporating prior data can reduce the time required to learn a range of robotic skills to practical time-scales.

arXiv comment17 pages. Website: https://awacrl.github.io/

References in corpus (21)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Reinforcement Learning from Imperfect Demonstrations
• Critic Regularized Regression
• Off-Policy Actor-Critic
• GenDICE: Generalized Offline Estimation of Stationary Values
• Consistent On-Line Off-Policy Evaluation
• Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies
• Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis
• Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
• Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models

Cited by in corpus (45)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• Review: Deep Learning in Electron Microscopy
• d3rlpy: An Offline Deep Reinforcement Learning Library
• COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Model-Based Visual Planning with Self-Supervised Functional Distances
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Heuristic-Guided Reinforcement Learning
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Offline Reinforcement Learning with Value-based Episodic Memory
• Offline Reinforcement Learning with Soft Behavior Regularization
• Demonstration-Guided Reinforcement Learning with Learned Skills
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
• Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• MUSBO: Model-based Uncertainty Regularized and Sample Efficient Batch Optimization for Deployment Constrained Reinforcement Learning
• Reinforcement Learning with Videos: Combining Offline Observations with Interaction
• AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scale
• Collect & Infer -- a fresh look at data-efficient Reinforcement Learning
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation
• Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
• Learning Off-Policy with Online Planning
• Dual Behavior Regularized Reinforcement Learning
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• Towards an Understanding of Default Policies in Multitask Policy Optimization
• Understanding the World Through Action
• Offline Inverse Reinforcement Learning
• Piano Fingering with Reinforcement Learning
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• Quantile Filtered Imitation Learning
• Solving the Real Robot Challenge using Deep Reinforcement Learning