Offline Meta-Reinforcement Learning with Advantage Weighting

Abstract

This paper introduces the offline meta-reinforcement learning (offline meta-RL) problem setting and proposes an algorithm that performs well in this setting. Offline meta-RL is analogous to the widely successful supervised learning strategy of pre-training a model on a large batch of fixed, pre-collected data (possibly from various tasks) and fine-tuning the model to a new task with relatively little data. That is, in offline meta-RL, we meta-train on fixed, pre-collected data from several tasks in order to adapt to a new task with a very small amount (less than 5 trajectories) of data from the new task. By nature of being offline, algorithms for offline meta-RL can utilize the largest possible pool of training data available and eliminate potentially unsafe or costly data collection during meta-training. This setting inherits the challenges of offline RL, but it differs significantly because offline RL does not generally consider a) transfer to new tasks or b) limited data from the test task, both of which we face in offline meta-RL. Targeting the offline meta-RL setting, we propose Meta-Actor Critic with Advantage Weighting (MACAW), an optimization-based meta-learning algorithm that uses simple, supervised regression objectives for both the inner and outer loop of meta-training. On offline variants of common meta-RL benchmarks, we empirically find that this approach enables fully offline meta-reinforcement learning and achieves notable gains over prior methods.

ICML 2021; for code & project info, see http://sites.google.com/view/macaw-metarl

References in corpus (16)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• One-Shot Visual Imitation Learning via Meta-Learning
• Behavior Regularized Offline Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Meta-Learning with Implicit Gradients
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Generalized Inner Loop Meta-Learning
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• When MAML Can Adapt Fast and How to Assist When It Cannot
• Offline Meta Learning of Exploration

Cited by in corpus (4)

• Offline Meta Learning of Exploration
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• Adaptation of Quadruped Robot Locomotion with Meta-Learning