Acceleration of Gradient-based Path Integral Method for Efficient Optimal and Inverse Optimal Control

Abstract

This paper deals with a new accelerated path integral method, which iteratively searches optimal controls with a small number of iterations. This study is based on the recent observations that a path integral method for reinforcement learning can be interpreted as gradient descent. This observation also applies to an iterative path integral method for optimal control, which sets a convincing argument for utilizing various optimization methods for gradient descent, such as momentum-based acceleration, step-size adaptation and their combination. We introduce these types of methods to the path integral and demonstrate that momentum-based methods, like Nesterov Accelerated Gradient and Adam, can significantly improve the convergence rate to search for optimal controls in simulated control systems. We also demonstrate that the accelerated path integral could improve the performance on model predictive control for various vehicle navigation tasks. Finally, we represent this accelerated path integral method as a recurrent network, which is the accelerated version of the previously proposed path integral networks (PI-Net). We can train the accelerated PI-Net more efficiently for inverse optimal control with less RAM than the original PI-Net.

ICRA2018 camera ready version

References in corpus (2)

• Path Integral Networks: End-to-End Differentiable Optimal Control
• Mirror Descent Search and its Acceleration

Cited by in corpus (7)

• Variational Inference MPC for Bayesian Model-based Reinforcement Learning
• Biased-MPPI: Informing Sampling-Based Model Predictive Control by Fusing Ancillary Controllers
• Learning to Optimize in Model Predictive Control
• PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference
• Real-time Sampling-based Model Predictive Control based on Reverse Kullback-Leibler Divergence and Its Adaptive Acceleration
• Control as Hybrid Inference
• Reinforcement Learning as Iterative and Amortised Inference