Offline Meta-Reinforcement Learning with Online Self-Supervision

Abstract

Meta-reinforcement learning (RL) methods can meta-train policies that adapt to new tasks with orders of magnitude less data than standard RL, but meta-training itself is costly and time-consuming. If we can meta-train on offline data, then we can reuse the same static dataset, labeled once with rewards for different tasks, to meta-train policies that adapt to a variety of new tasks at meta-test time. Although this capability would make meta-RL a practical tool for real-world use, offline meta-RL presents additional challenges beyond online meta-RL or standard offline RL settings. Meta-RL learns an exploration strategy that collects data for adapting, and also meta-trains a policy that quickly adapts to data from a new task. Since this policy was meta-trained on a fixed, offline dataset, it might behave unpredictably when adapting to data collected by the learned exploration strategy, which differs systematically from the offline data and thus induces distributional shift. We propose a hybrid offline meta-RL algorithm, which uses offline data with rewards to meta-train an adaptive policy, and then collects additional unsupervised online data, without any reward labels to bridge this distribution shift. By not requiring reward labels for online collection, this data can be much cheaper to collect. We compare our method to prior work on offline meta-RL on simulated robot locomotion and manipulation tasks and find that using additional unsupervised online data collection leads to a dramatic improvement in the adaptive capabilities of the meta-trained policies, matching the performance of fully online meta-RL on a range of challenging domains that require generalization to new tasks.

8.5 pages, 6 figures, accepted to ICML 2022

References in corpus (18)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Emergence of Locomotion Behaviours in Rich Environments
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Offline Learning from Demonstrations and Unlabeled Experience
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Positive-Unlabeled Reward Learning
• Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization
• Offline Meta Learning of Exploration
• Semi-supervised reward learning for offline reinforcement learning
• Disentangled Cumulants Help Successor Representations Transfer to New Tasks
• MELD: Meta-Reinforcement Learning from Images via Latent State Models