Meta Reinforcement Learning with Task Embedding and Shared Policy

Abstract

Despite significant progress, deep reinforcement learning (RL) suffers from data-inefficiency and limited generalization. Recent efforts apply meta-learning to learn a meta-learner from a set of RL tasks such that a novel but related task could be solved quickly. Though specific in some ways, different tasks in meta-RL are generally similar at a high level. However, most meta-RL methods do not explicitly and adequately model the specific and shared information among different tasks, which limits their ability to learn training tasks and to generalize to novel tasks. In this paper, we propose to capture the shared information on the one hand and meta-learn how to quickly abstract the specific information about a task on the other hand. Methodologically, we train an SGD meta-learner to quickly optimize a task encoder for each task, which generates a task embedding based on past experience. Meanwhile, we learn a policy which is shared across all tasks and conditioned on task embeddings. Empirical results on four simulated tasks demonstrate that our method has better learning capacity on both training and novel tasks and attains up to 3 to 4 times higher returns compared to baselines.

Accepted to IJCAI 2019

References in corpus (13)

• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Playing Atari with Deep Reinforcement Learning
• An Overview of Multi-Task Learning in Deep Neural Networks
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Evolved Policy Gradients
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Unsupervised Meta-Learning for Reinforcement Learning
• Learning to Explore with Meta-Policy Gradient

Cited by in corpus (6)

• Multi-Task Learning with Deep Neural Networks: A Survey
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
• Model-based Adversarial Meta-Reinforcement Learning
• Multi-task Batch Reinforcement Learning with Metric Learning
• Neural Stochastic Dual Dynamic Programming
• Connecting Context-specific Adaptation in Humans to Meta-learning