Variational Inference MPC for Bayesian Model-based Reinforcement Learning

Abstract

In recent studies on model-based reinforcement learning (MBRL), incorporating uncertainty in forward dynamics is a state-of-the-art strategy to enhance learning performance, making MBRLs competitive to cutting-edge model free methods, especially in simulated robotics tasks. Probabilistic ensembles with trajectory sampling (PETS) is a leading type of MBRL, which employs Bayesian inference to dynamics modeling and model predictive control (MPC) with stochastic optimization via the cross entropy method (CEM). In this paper, we propose a novel extension to the uncertainty-aware MBRL. Our main contributions are twofold: Firstly, we introduce a variational inference MPC, which reformulates various stochastic methods, including CEM, in a Bayesian fashion. Secondly, we propose a novel instance of the framework, called probabilistic action ensembles with trajectory sampling (PaETS). As a result, our Bayesian MBRL can involve multimodal uncertainties both in dynamics and optimal trajectories. In comparison to PETS, our method consistently improves asymptotic performance on several challenging locomotion tasks.

arXiv commentAccepted to CoRL2019. Camera-ready ver

References in corpus (10)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• MCMC using Hamiltonian dynamics
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Learning Latent Dynamics for Planning from Pixels
• Model-Ensemble Trust-Region Policy Optimization
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Acceleration of Gradient-based Path Integral Method for Efficient Optimal and Inverse Optimal Control
• Mirror Descent Search and its Acceleration

Cited by in corpus (14)

• Reinforcement Learning through Active Inference
• Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization
• Constrained Model-based Reinforcement Learning with Robust Cross-Entropy Method
• Model-Predictive Control via Cross-Entropy and Gradient-Based Optimization
• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
• The Differentiable Cross-Entropy Method
• The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning
• Control as Hybrid Inference
• Domain-Adversarial and Conditional State Space Model for Imitation Learning
• Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
• Reinforcement Learning as Iterative and Amortised Inference
• Bayes-Adaptive Deep Model-Based Policy Optimisation
• Multi-person Pose Tracking using Sequential Monte Carlo with Probabilistic Neural Pose Predictor
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning