Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning

Abstract

Model-based reinforcement learning (RL) has shown great potential in various control tasks in terms of both sample-efficiency and final performance. However, learning a generalizable dynamics model robust to changes in dynamics remains a challenge since the target transition dynamics follow a multi-modal distribution. In this paper, we present a new model-based RL algorithm, coined trajectory-wise multiple choice learning, that learns a multi-headed dynamics model for dynamics generalization. The main idea is updating the most accurate prediction head to specialize each head in certain environments with similar dynamics, i.e., clustering environments. Moreover, we incorporate context learning, which encodes dynamics-specific information from past experiences into the context latent vector, enabling the model to perform online adaptation to unseen environments. Finally, to utilize the specialized prediction heads more effectively, we propose an adaptive planning method, which selects the most accurate prediction head over a recent experience. Our method exhibits superior zero-shot generalization performance across a variety of control tasks, compared to state-of-the-art RL methods. Source code and videos are available at https://sites.google.com/view/trajectory-mcl.

Accepted in NeurIPS2020. First two authors contributed equally, website: https://sites.google.com/view/trajectory-mcl code: https://github.com/younggyoseo/trajectory_mcl

References in corpus (13)

• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Learning Latent Dynamics for Planning from Pixels
• Graph networks as learnable physics engines for inference and control
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• When to Trust Your Model: Model-Based Policy Optimization
• Assessing Generalization in Deep Reinforcement Learning
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloads
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning

Cited by in corpus (3)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments