Path integrals and symmetry breaking for optimal control theory

Abstract

This paper considers linear-quadratic control of a non-linear dynamical system subject to arbitrary cost. I show that for this class of stochastic control problems the non-linear Hamilton-Jacobi-Bellman equation can be transformed into a linear equation. The transformation is similar to the transformation used to relate the classical Hamilton-Jacobi equation to the Schrödinger equation. As a result of the linearity, the usual backward computation can be replaced by a forward diffusion process, that can be computed by stochastic integration or by the evaluation of a path integral. It is shown, how in the deterministic limit the PMP formalism is recovered. The significance of the path integral approach is that it forms the basis for a number of efficient computational methods, such as MC sampling, the Laplace approximation and the variational approximation. We show the effectiveness of the first two methods in number of examples. Examples are given that show the qualitative difference between stochastic and deterministic control and the occurrence of symmetry breaking as a function of the noise.

21 pages, 6 figures, submitted to JSTAT

Cited by in corpus (86)

• A Brief Survey of Deep Reinforcement Learning
• Soft Actor-Critic Algorithms and Applications
• Reinforcement Learning with Deep Energy-Based Policies
• A free energy principle for a particular physics
• Maximum a Posteriori Policy Optimisation
• The free energy principle made simpler but not too simple
• Variational and optimal control representations of conditioned and driven processes
• Path integrals, particular kinds, and strange things
• Neural Network-based Flight Control Systems: Present and Future
• Active Learning in Robotics: A Review of Control Principles
• The value of monitoring to control evolving populations
• Learning Optimal Control via Forward and Backward Stochastic Differential Equations
• Latent Space Policies for Hierarchical Reinforcement Learning
• Path Integral Control and State Dependent Feedback
• Efficient Rare Event Simulation by Optimal Nonequilibrium Forcing
• Uncertainty Averse Pushing with Model Predictive Path Integral Control
• If MaxEnt RL is the Answer, What is the Question?
• Biased-MPPI: Informing Sampling-Based Model Predictive Control by Fusing Ancillary Controllers
• Limited-control optimal protocols arbitrarily far from equilibrium
• Learning to be Safe: Deep RL with a Safety Critic
• Mirror Descent Policy Optimization
• Variational approach to rare event simulation using least-squares regression
• Optimal navigation of microswimmers in complex and noisy environments
• Approximate Inference and Stochastic Optimal Control
• Stochastic Optimal Control as Non-equilibrium Statistical Mechanics: Calculus of Variations over Density and Current
• Stein Variational Model Predictive Control
• Black-Box Policy Search with Probabilistic Programs
• Effects of water currents on fish migration through a Feynman-type path integral approach under $\sqrt{8/3}$ Liouville-like quantum gravity surfaces
• Real-Time Stochastic Optimal Control for Multi-agent Quadrotor Systems
• Multi-Agent Path Integral Control for Interaction-Aware Motion Planning in Urban Canals
• Shaping Individualized Impedance Landscapes for Gait Training via Reinforcement Learning
• Sample Efficient Path Integral Control under Uncertainty
• Probabilistic design of optimal sequential decision-making algorithms in learning and control
• Schrödinger Approach to Optimal Control of Large-Size Populations
• Energetic cost of microswimmer navigation: the role of body shape
• Linearly Solvable Stochastic Control Lyapunov Functions
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Pontryagin's Minimum Principle and Forward-Backward Sweep Method for the System of HJB-FP Equations in Memory-Limited Partially Observable Stochastic Control
• Divergence-Augmented Policy Optimization
• Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework
• Optimality and Suboptimality of MPPI Control in Stochastic and Deterministic Settings
• C-Learning: Learning to Achieve Goals via Recursive Classification
• Topology-Guided Path Integral Approach for Stochastic Optimal Control in Cluttered Environment
• Thinking While Moving: Deep Reinforcement Learning with Concurrent Control
• Disentangled Skill Embeddings for Reinforcement Learning
• Optimal evolutionary control for artificial selection on molecular phenotypes
• Path Consistency Learning in Tsallis Entropy Regularized MDPs
• Optimal Control in Soft and Active Matter
• Local Search for Policy Iteration in Continuous Control
• Disordered high-dimensional optimal control
• A Regularized Opponent Model with Maximum Entropy Objective
• On Kalman-Bucy filters, linear quadratic control and active inference
• A Multilevel Approach for Stochastic Nonlinear Optimal Control
• Model-based Path Integral Stochastic Control: A Bayesian Nonparametric Approach
• Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control
• Active Inference or Control as Inference? A Unifying View
• POMDPs in Continuous Time and Discrete Spaces
• Making Birth-Death Processes from Backward Fokker-Planck Equations for Computing Expectations in Langevin Systems
• Decision Making under Uncertainty: A Quasimetric Approach
• A fictitious-play finite-difference method for linearly solvable mean field games
• Dynamic Policy Programming
• Meshless discretization of LQ-type stochastic control problems
• Multitime stochastic maximum principle on curvilinear integral actions
• Path integral formulation of stochastic optimal control with generalized costs
• Variational Inference for Stochastic Control of Infinite Dimensional Systems
• Deep Forward-Backward SDEs for Min-max Control
• Harmonic Path Integral Diffusion
• Connection among stochastic Hamilton-Jacobi-Bellman equation, path-integral, and Koopman operator on nonlinear stochastic optimal control
• Nonlinear stochastic receding horizon control: stability, robustness and Monte Carlo methods for control approximation
• Noise equals control
• Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
• Noncooperative dynamics in election interference
• Deep PQR: Solving Inverse Reinforcement Learning using Anchor Actions
• Iterative control strategies for non-linear systems
• Optimal control of stochastic reaction networks with entropic control cost and emergence of mode-switching strategies
• Regularized Policies are Reward Robust
• Semidefinite Relaxations for Stochastic Optimal Control Policies
• Bounded Rational Decision-Making in Changing Environments
• MPPI-VS: Sampling-Based Model Predictive Control Strategy for Constrained Image-Based and Position-Based Visual Servoing
• Bayesian Fusion of Data Partitioned Particle Estimates
• Stochastic quantization approach to modeling of constrained Ito processes in discrete time
• Model-Free Risk-Sensitive Reinforcement Learning
• Deterministic particle flows for constraining stochastic nonlinear systems
• Stochastic optimization for learning quantum state feedback control
• Nonparametric Infinite Horizon Kullback-Leibler Stochastic Control
• Hierarchical model-based policy optimization: from actions to action sequences and back