Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future

Abstract

In model-based reinforcement learning, the agent interleaves between model learning and planning. These two components are inextricably intertwined. If the model is not able to provide sensible long-term prediction, the executed planner would exploit model flaws, which can yield catastrophic failures. This paper focuses on building a model that reasons about the long-term future and demonstrates how to use this for efficient planning and exploration. To this end, we build a latent-variable autoregressive model by leveraging recent ideas in variational inference. We argue that forcing latent variables to carry future information through an auxiliary task substantially improves long-term predictions. Moreover, by planning in the latent space, the planner's solution is ensured to be within regions where the model is valid. An exploration strategy can be devised by searching for unlikely trajectories under the model. Our method achieves higher reward faster compared to baselines on a variety of tasks and environments in both the imitation learning and model-based reinforcement learning settings.

To appear at ICLR 2019

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Trust Region Policy Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• A Recurrent Latent Variable Model for Sequential Data
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Learning Stochastic Recurrent Networks
• Deep Kalman Filters
• Recurrent Environment Simulators
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Lipschitz Continuity in Model-based Reinforcement Learning
• The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces
• Prediction and Control with Temporal Segment Models

Cited by in corpus (7)

• Meta reinforcement learning as task inference
• Objective Mismatch in Model-based Reinforcement Learning
• Variational Recurrent Models for Solving Partially Observable Control Tasks
• Improving Robot Dual-System Motor Learning with Intrinsically Motivated Meta-Control and Latent-Space Experience Imagination
• Active World Model Learning with Progress Curiosity
• Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction
• Episodic Memory for Learning Subjective-Timescale Models