Meta-Q-Learning

Abstract

This paper introduces Meta-Q-Learning (MQL), a new off-policy algorithm for meta-Reinforcement Learning (meta-RL). MQL builds upon three simple ideas. First, we show that Q-learning is competitive with state-of-the-art meta-RL algorithms if given access to a context variable that is a representation of the past trajectory. Second, a multi-task objective to maximize the average reward across the training tasks is an effective method to meta-train RL policies. Third, past data from the meta-training replay buffer can be recycled to adapt the policy on a new task using off-policy updates. MQL draws upon ideas in propensity estimation to do so and thereby amplifies the amount of available data for adaptation. Experiments on standard continuous-control benchmarks suggest that MQL compares favorably with the state of the art in meta-RL.

arXiv commentICLR 2020 conference paper

Cited by in corpus (11)

• Learning to Learn Variational Semantic Memory
• Model-based Adversarial Meta-Reinforcement Learning
• Multi-task Batch Reinforcement Learning with Metric Learning
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• Neural Stochastic Dual Dynamic Programming
• Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning
• Offline Meta Learning of Exploration
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• Improved Robustness and Safety for Pre-Adaptation of Meta Reinforcement Learning with Prior Regularization