Meta-Model-Based Meta-Policy Optimization

Abstract

Model-based meta-reinforcement learning (RL) methods have recently been shown to be a promising approach to improving the sample efficiency of RL in multi-task settings. However, the theoretical understanding of those methods is yet to be established, and there is currently no theoretical guarantee of their performance in a real-world environment. In this paper, we analyze the performance guarantee of model-based meta-RL methods by extending the theorems proposed by Janner et al. (2019). On the basis of our theoretical results, we propose Meta-Model-Based Meta-Policy Optimization (M3PO), a model-based meta-RL method with a performance guarantee. We demonstrate that M3PO outperforms existing meta-RL methods in continuous-control benchmarks.

arXiv commentACML 2021. Video demo: https://drive.google.com/file/d/1DRA-pmIWnHGNv5G_gFrml8YzKCtMcGnu/view?usp=sharing URL Source code: https://github.com/TakuyaHiraoka/Meta-Model-Based-Meta-Policy-Optimization

References in corpus (23)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• A Simple Neural Attentive Meta-Learner
• Searching for Activation Functions
• Learning to reinforcement learn
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Meta-Reinforcement Learning of Structured Exploration Strategies
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Model Predictive Path Integral Control using Covariance Variable Importance Sampling
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Meta reinforcement learning as task inference
• A Game Theoretic Framework for Model Based Reinforcement Learning
• Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
• Guided Meta-Policy Search
• Deep Variational Reinforcement Learning for POMDPs
• Explicit Explore-Exploit Algorithms in Continuous State Spaces

Cited by in corpus (4)

• Dropout Q-Functions for Doubly Efficient Reinforcement Learning
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models
• Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments