Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learning
arXiv:1705.00470
Abstract
In this paper we study how to learn stochastic, multimodal transition dynamics in reinforcement learning (RL) tasks. We focus on evaluating transition function estimation, while we defer planning over this model to future work. Stochasticity is a fundamental property of many task environments. However, discriminative function approximators have difficulty estimating multimodal stochasticity. In contrast, deep generative models do capture complex high-dimensional outcome distributions. First we discuss why, amongst such models, conditional variational inference (VI) is theoretically most appealing for model-based RL. Subsequently, we compare different VI models on their ability to learn complex stochasticity on simulated functions, as well as on a typical RL gridworld with multimodal dynamics. Results show VI successfully predicts multimodal outcomes, but also robustly ignores these for deterministic parts of the transition dynamics. In summary, we show a robust method to learn multimodal transitions using function approximation, which is a key preliminary for model-based RL in stochastic domains.
Scaling Up Reinforcement Learning (SURL) Workshop @ European Machine Learning Conference (ECML)
References in corpus (3)
Cited by in corpus (9)
- Unsupervised Video Object Segmentation for Deep Reinforcement Learning
- Lipschitz Continuity in Model-based Reinforcement Learning
- Combating the Compounding-Error Problem with a Multi-step Model
- Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables
- Bayesian Semisupervised Learning with Deep Generative Models
- Efficient exploration with Double Uncertain Value Networks
- The Potential of the Return Distribution for Exploration in RL
- Variational Inference for Data-Efficient Model Learning in POMDPs
- Evidential Sparsification of Multimodal Latent Spaces in Conditional Variational Autoencoders