Approximate Inference and Stochastic Optimal Control

Abstract

We propose a novel reformulation of the stochastic optimal control problem as an approximate inference problem, demonstrating, that such a interpretation leads to new practical methods for the original problem. In particular we characterise a novel class of iterative solutions to the stochastic optimal control problem based on a natural relaxation of the exact dual formulation. These theoretical insights are applied to the Reinforcement Learning problem where they lead to new model free, off policy methods for discrete and continuous problems.

References in corpus (3)

• Path integrals and symmetry breaking for optimal control theory
• Optimal control as a graphical model inference problem
• Dynamic Policy Programming

Cited by in corpus (11)

• Continuous Deep Q-Learning with Model-based Acceleration
• Taming the Noise in Reinforcement Learning via Soft Updates
• VIREL: A Variational Inference Framework for Reinforcement Learning
• Action and Perception as Divergence Minimization
• Stein Variational Model Predictive Control
• Entropy Regularized Motion Planning via Stein Variational Inference
• A Nonparametric Conjugate Prior Distribution for the Maximizing Argument of a Noisy Function
• Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
• Joint Perception and Control as Inference with an Object-based Implementation
• Parameterized MDPs and Reinforcement Learning Problems -- A Maximum Entropy Principle Based Framework
• Count-Based Temperature Scheduling for Maximum Entropy Reinforcement Learning