On Entropy Regularized Path Integral Control for Trajectory Optimization

Abstract

In this article we present a generalised view on Path Integral Control (PIC) methods. PIC refers to a particular class of policy search methods that are closely tied to the setting of Linearly Solvable Optimal Control (LSOC), a restricted subclass of nonlinear Stochastic Optimal Control (SOC) problems. This class is unique in the sense that it can be solved explicitly to yield a formal optimal state trajectory distribution. In this contribution we first review the PIC theory and discuss related algorithms tailored to policy search in general. We are able to identify a generic design strategy that relies on the existence of an optimal state trajectory distribution and finds a parametric policy by minimizing the cross entropy between the optimal and a state trajectory distribution parametrized through its policy. Inspired by this observation we then aim to formulate a SOC problem that shares traits with the LSOC setting yet that covers a less restrictive class of problem formulations. We refer to this SOC problem as Entropy Regularized Trajectory Optimization. The problem is closely related to the Entropy Regularized Stochastic Optimal Control setting which is lately often addressed by the Reinforcement Learning (RL) community. We analyse the theoretical convergence behaviour of the theoretical state trajectory distribution sequence and draw connections with stochastic search methods tailored to classic optimization problems. Finally we derive explicit updates and compare the implied Entropy Regularized PIC with earlier work in the context of both PIC and RL for derivative-free trajectory optimization.

References in corpus (21)

• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Emergence of Locomotion Behaviours in Rich Environments
• The CMA Evolution Strategy: A Tutorial
• Reinforcement Learning with Deep Energy-Based Policies
• A linear theory for control of non-linear stochastic systems
• CMA-ES for Hyperparameter Optimization of Deep Neural Networks
• Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles
• Path Integral Policy Improvement with Covariance Matrix Adaptation
• A unified view of entropy-regularized Markov decision processes
• Adaptive importance sampling for control and inference
• Updating Probabilities with Data and Moments
• Entropic Inference
• Model-Free Trajectory-based Policy Optimization with Monotonic Improvement
• Sample Efficient Path Integral Control under Uncertainty
• Importance mixing: Improving sample reuse in evolutionary policy search methods
• Entropic Inference: some pitfalls and paradoxes we can avoid
• Lyceum: An efficient and scalable ecosystem for robot learning
• Adaptive Smoothing Path Integral Control
• Minima distribution for global optimization

Cited by in corpus (1)

• Risk Sensitive Path Integral Control for Infinite Horizon Problem Formulations