Adaptive Smoothing Path Integral Control
arXiv:2005.06364
Abstract
In Path Integral control problems a representation of an optimally controlled dynamical system can be formally computed and serve as a guidepost to learn a parametrized policy. The Path Integral Cross-Entropy (PICE) method tries to exploit this, but is hampered by poor sample efficiency. We propose a model-free algorithm called ASPIC (Adaptive Smoothing of Path Integral Control) that applies an inf-convolution to the cost function to speedup convergence of policy optimization. We identify PICE as the infinite smoothing limit of such technique and show that the sample efficiency problems that PICE suffers disappear for finite levels of smoothing. For zero smoothing this method becomes a greedy optimization of the cost, which is the standard approach in current reinforcement learning. We show analytically and empirically that intermediate levels of smoothing are optimal, which renders the new method superior to both PICE and direct cost-optimization.
arXiv comment23 pages, 5 figures, NeurIPS 2019 Optimization Foundations of Reinforcement Learning Workshop (OptRL 2019)
References in corpus (9)
- Trust Region Policy Optimization
- Emergence of Locomotion Behaviours in Rich Environments
- A linear theory for control of non-linear stochastic systems
- Optimal control as a graphical model inference problem
- New insights and perspectives on the natural gradient method
- Adaptive importance sampling for control and inference
- Path Integral Control and State Dependent Feedback
- Explicit solution of relative entropy weighted control
- Particle Smoothing for Hidden Diffusion Processes: Adaptive Path Integral Smoother