Model-based Adversarial Meta-Reinforcement Learning

Abstract

Meta-reinforcement learning (meta-RL) aims to learn from multiple training tasks the ability to adapt efficiently to unseen test tasks. Despite the success, existing meta-RL algorithms are known to be sensitive to the task distribution shift. When the test task distribution is different from the training task distribution, the performance may degrade significantly. To address this issue, this paper proposes Model-based Adversarial Meta-Reinforcement Learning (AdMRL), where we aim to minimize the worst-case sub-optimality gap -- the difference between the optimal return and the return that the algorithm achieves after adaptation -- across all tasks in a family of tasks, with a model-based approach. We propose a minimax objective and optimize it by alternating between learning the dynamics model on a fixed task and finding the adversarial task for the current model -- the task for which the policy induced by the model is maximally suboptimal. Assuming the family of tasks is parameterized, we derive a formula for the gradient of the suboptimality with respect to the task parameters via the implicit function theorem, and show how the gradient estimator can be efficiently implemented by the conjugate gradient method and a novel use of the REINFORCE estimator. We evaluate our approach on several continuous control benchmarks and demonstrate its efficacy in the worst-case performance over all tasks, the generalization power to out-of-distribution tasks, and in training and test time sample efficiency, over existing state-of-the-art meta-RL algorithms.

Accepted by NeurIPS 2020. Code at https://github.com/LinZichuan/AdMRL

References in corpus (28)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Playing Atari with Deep Reinforcement Learning
• Theoretical Models of Learning to Learn
• Learning to reinforcement learn
• Large-Scale Study of Curiosity-Driven Learning
• Model-Ensemble Trust-Region Policy Optimization
• Deep Equilibrium Models
• Exploration by Random Network Distillation
• Benchmarking Model-Based Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Exploring Model-based Planning with Policy Networks
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Meta reinforcement learning as task inference
• A Game Theoretic Framework for Model Based Reinforcement Learning
• Unsupervised Meta-Learning for Reinforcement Learning
• Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
• Guided Meta-Policy Search
• Meta-Q-Learning
• Reward-Free Exploration for Reinforcement Learning
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
• Meta Reinforcement Learning with Task Embedding and Shared Policy
• A Model-based Approach for Sample-efficient Multi-task Reinforcement Learning

Cited by in corpus (6)

• Challenges of Applying Deep Reinforcement Learning in Dynamic Dispatching
• Exploring Electron Beam Induced Atomic Assembly via Reinforcement Learning in a Molecular Dynamics Environment
• Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments
• Improved Robustness and Safety for Pre-Adaptation of Meta Reinforcement Learning with Prior Regularization