A Model-based Approach for Sample-efficient Multi-task Reinforcement Learning

Abstract

The aim of multi-task reinforcement learning is two-fold: (1) efficiently learn by training against multiple tasks and (2) quickly adapt, using limited samples, to a variety of new tasks. In this work, the tasks correspond to reward functions for environments with the same (or similar) dynamical models. We propose to learn a dynamical model during the training process and use this model to perform sample-efficient adaptation to new tasks at test time. We use significantly fewer samples by performing policy optimization only in a "virtual" environment whose transitions are given by our learned dynamical model. Our algorithm sequentially trains against several tasks. Upon encountering a new task, we first warm-up a policy on our learned dynamical model, which requires no new samples from the environment. We then adapt the dynamical model with samples from this policy in the real environment. We evaluate our approach on several continuous control benchmarks and demonstrate its efficacy over MAML, a state-of-the-art meta-learning algorithm, on these tasks.

arXiv comment13 pages, 3 figures

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Trust Region Policy Optimization
• Theoretical Models of Learning to Learn
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Model-Based Reinforcement Learning for Atari
• Learning to reinforcement learn
• Model-Ensemble Trust-Region Policy Optimization
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm

Cited by in corpus (6)

• Solving Rubik's Cube with a Robot Hand
• Model-based Adversarial Meta-Reinforcement Learning
• Fractional Transfer Learning for Deep Model-Based Reinforcement Learning
• Provably Efficient Multi-Task Reinforcement Learning with Model Transfer
• Robot in a China Shop: Using Reinforcement Learning for Location-Specific Navigation Behaviour
• On The Transferability of Deep-Q Networks