Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling

Abstract

Reinforcement learning algorithms can acquire policies for complex tasks autonomously. However, the number of samples required to learn a diverse set of skills can be prohibitively large. While meta-reinforcement learning methods have enabled agents to leverage prior experience to adapt quickly to new tasks, their performance depends crucially on how close the new task is to the previously experienced tasks. Current approaches are either not able to extrapolate well, or can do so at the expense of requiring extremely large amounts of data for on-policy meta-training. In this work, we present model identification and experience relabeling (MIER), a meta-reinforcement learning algorithm that is both efficient and extrapolates well when faced with out-of-distribution tasks at test time. Our method is based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data, more easily than policies and value functions. These dynamics models can then be used to continue training policies and value functions for out-of-distribution tasks without using meta-reinforcement learning at all, by generating synthetic experience for the new task.

References in corpus (15)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning to reinforcement learn
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Meta-Learning with Latent Embedding Optimization
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Evolved Policy Gradients
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• Model Predictive Path Integral Control using Covariance Variable Importance Sampling
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Meta reinforcement learning as task inference
• Guided Meta-Policy Search
• Meta Reinforcement Learning with Task Embedding and Shared Policy

Cited by in corpus (9)

• Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environments
• Model-based Adversarial Meta-Reinforcement Learning
• Linear Representation Meta-Reinforcement Learning for Instant Adaptation
• Adaptive Adversarial Training for Meta Reinforcement Learning
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• Causal Multi-Agent Reinforcement Learning: Review and Open Problems
• Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments
• On the Practical Consistency of Meta-Reinforcement Learning Algorithms