Adaptive importance sampling for control and inference
arXiv:1505.01874 · doi:10.1007/s10955-016-1446-7
Abstract
Path integral (PI) control problems are a restricted class of non-linear control problems that can be solved formally as a Feyman-Kac path integral and can be estimated using Monte Carlo sampling. In this contribution we review path integral control theory in the finite horizon case. We subsequently focus on the problem how to compute and represent control solutions. Within the PI theory, the question of how to compute becomes the question of importance sampling. Efficient importance samplers are state feedback controllers and the use of these requires an efficient representation. Learning and representing effective state-feedback controllers for non-linear stochastic control problems is a very challenging, and largely unsolved, problem. We show how to learn and represent such controllers using ideas from the cross entropy method. We derive a gradient descent method that allows to learn feed-back controllers using an arbitrary parametrisation. We refer to this method as the Path Integral Cross Entropy method or PICE. We illustrate this method for some simple examples. The path integral control methods can be used to estimate the posterior distribution in latent state models. In neuroscience these problems arise when estimating connectivity from neural recording data using EM. We demonstrate the path integral control method as an accurate alternative to particle filtering.
23 pages, 4 figures
References in corpus (2)
Cited by in corpus (36)
- Scalable Gradients for Stochastic Differential Equations
- A geometric approach to optimal nonequilibrium control: Minimizing dissipation in nanomagnetic spin systems
- Near-optimal protocols in complex nonequilibrium transformations
- A reinforcement learning approach to rare trajectory sampling
- Adaptive sampling of large deviations
- Data Assimilation: The Schrödinger Perspective
- Implicitly Adaptive Importance Sampling
- Reinforcement learning of rare diffusive dynamics
- Learning to Guide: Guidance Law Based on Deep Meta-learning and Model Predictive Path Integral Control
- Controlled Sequential Monte Carlo
- Particle Smoothing for Hidden Diffusion Processes: Adaptive Path Integral Smoother
- Evolutionary reinforcement learning of dynamical large deviations
- Variational approach to rare event simulation using least-squares regression
- Deep Value Model Predictive Control
- An Optimal Control Derivation of Nonlinear Smoothing Equations
- Rare Events via Cross-Entropy Population Monte Carlo
- Schrödinger Bridge Samplers
- Adaptive Path-Integral Autoencoder: Representation Learning and Planning for Dynamical Systems
- Optimal evolutionary control for artificial selection on molecular phenotypes
- Topology-Guided Path Integral Approach for Stochastic Optimal Control in Cluttered Environment
- On Entropy Regularized Path Integral Control for Trajectory Optimization
- A Multilevel Approach for Stochastic Nonlinear Optimal Control
- Annealed Flow Transport Monte Carlo
- Consistent Adaptive Multiple Importance Sampling and Controlled Diffusions
- On Unbiased Score Estimation for Partially Observed Diffusions
- Action selection in growing state spaces: Control of Network Structure Growth
- Adaptive Smoothing Path Integral Control
- Harmonic Path Integral Diffusion
- Deterministic particle flows for constraining SDEs
- Backward Simulation of Stochastic Process using a Time Reverse Monte Carlo method
- Reinforcement Learning as Iterative and Amortised Inference
- Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning
- Deterministic particle flows for constraining stochastic nonlinear systems
- Duality for nonlinear filtering
- Stochastic optimization for learning quantum state feedback control
- A Topology-Guided Path Integral Approach for Stochastic Optimal Control