Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning

Abstract

Model-based reinforcement learning (RL) enjoys several benefits, such as data-efficiency and planning, by learning a model of the environment's dynamics. However, learning a global model that can generalize across different dynamics is a challenging task. To tackle this problem, we decompose the task of learning a global dynamics model into two stages: (a) learning a context latent vector that captures the local dynamics, then (b) predicting the next state conditioned on it. In order to encode dynamics-specific information into the context latent vector, we introduce a novel loss function that encourages the context latent vector to be useful for predicting both forward and backward dynamics. The proposed method achieves superior generalization ability across various simulated robotics and control tasks, compared to existing RL schemes.

Accepted in ICML2020. First two authors contributed equally, website: https://sites.google.com/view/cadm code: https://github.com/younggyoseo/CaDM

References in corpus (13)

• Searching for Activation Functions
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Robust Adversarial Reinforcement Learning
• Deep Reinforcement Learning that Matters
• Learning Latent Dynamics for Planning from Pixels
• Continuous Deep Q-Learning with Model-based Acceleration
• Graph networks as learnable physics engines for inference and control
• Benchmarking Model-Based Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Assessing Generalization in Deep Reinforcement Learning
• Environment Probing Interaction Policies

Cited by in corpus (4)

• Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
• Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap
• Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models