PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference

Abstract

In the present paper, we propose an extension of the Deep Planning Network (PlaNet), also referred to as PlaNet of the Bayesians (PlaNet-Bayes). There has been a growing demand in model predictive control (MPC) in partially observable environments in which complete information is unavailable because of, for example, lack of expensive sensors. PlaNet is a promising solution to realize such latent MPC, as it is used to train state-space models via model-based reinforcement learning (MBRL) and to conduct planning in the latent space. However, recent state-of-the-art strategies mentioned in MBRR literature, such as involving uncertainty into training and planning, have not been considered, significantly suppressing the training performance. The proposed extension is to make PlaNet uncertainty-aware on the basis of Bayesian inference, in which both model and action uncertainty are incorporated. Uncertainty in latent models is represented using a neural network ensemble to approximately infer model posteriors. The ensemble of optimal action candidates is also employed to capture multimodal uncertainty in the optimality. The concept of the action ensemble relies on a general variational inference MPC (VI-MPC) framework and its instance, probabilistic action ensemble with trajectory sampling (PaETS). In this paper, we extend VI-MPC and PaETS, which have been originally introduced in previous literature, to address partially observable cases. We experimentally compare the performances on continuous control tasks, and conclude that our method can consistently improve the asymptotic performance compared with PlaNet.

References in corpus (14)

• Auto-Encoding Variational Bayes
• DeepMind Control Suite
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Learning Latent Dynamics for Planning from Pixels
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Deep Dynamics Models for Learning Dexterous Manipulation
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Path Integral Networks: End-to-End Differentiable Optimal Control
• Acceleration of Gradient-based Path Integral Method for Efficient Optimal and Inverse Optimal Control
• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• Disentangling Controllable and Uncontrollable Factors of Variation by Interacting with the World
• Domain-Adversarial and Conditional State Space Model for Imitation Learning

Cited by in corpus (5)

• Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning
• Action and Perception as Divergence Minimization
• Domain-Adversarial and Conditional State Space Model for Imitation Learning
• Reinforcement Learning as Iterative and Amortised Inference
• Predictive Control Using Learned State Space Models via Rolling Horizon Evolution