Guided Meta-Policy Search

Abstract

Reinforcement learning (RL) algorithms have demonstrated promising results on complex tasks, yet often require impractical numbers of samples since they learn from scratch. Meta-RL aims to address this challenge by leveraging experience from previous tasks so as to more quickly solve new tasks. However, in practice, these algorithms generally also require large amounts of on-policy experience during the meta-training process, making them impractical for use in many problems. To this end, we propose to learn a reinforcement learning procedure in a federated way, where individual off-policy learners can solve the individual meta-training tasks, and then consolidate these solutions into a single meta-learner. Since the central meta-learner learns by imitating the solutions to the individual tasks, it can accommodate either the standard meta-RL problem setting or a hybrid setting where some or all tasks are provided with example demonstrations. The former results in an approach that can leverage policies learned for previous tasks without significant amounts of on-policy data during meta-training, whereas the latter is particularly useful in cases where demonstrations are easy for a person to provide. Across a number of continuous control meta-RL problems, we demonstrate significant improvements in meta-RL sample efficiency in comparison to prior work as well as the ability to scale to domains with visual observations.

arXiv commentPublished at Neurips 2019. Website : https://sites.google.com/berkeley.edu/guided-metapolicy-search

Cited by in corpus (15)

• Meta-Learning with Warped Gradient Descent
• Meta reinforcement learning as task inference
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Dropout Q-Functions for Doubly Efficient Reinforcement Learning
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
• Model-based Adversarial Meta-Reinforcement Learning
• A Meta-Reinforcement Learning Approach to Process Control
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Meta-Learning surrogate models for sequential decision making
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Meta-Model-Based Meta-Policy Optimization
• How Does the Task Landscape Affect MAML Performance?
• On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning
• Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces
• Meta Learning in the Continuous Time Limit