Path Integral Policy Improvement with Covariance Matrix Adaptation

Abstract

There has been a recent focus in reinforcement learning on addressing continuous state and action problems by optimizing parameterized policies. PI2 is a recent example of this approach. It combines a derivation from first principles of stochastic optimal control with tools from statistical estimation theory. In this paper, we consider PI2 as a member of the wider family of methods which share the concept of probability-weighted averaging to iteratively update parameters to optimize a cost function. We compare PI2 to other members of the same family - Cross-Entropy Methods and CMAES - at the conceptual level and in terms of performance. The comparison suggests the derivation of a novel algorithm which we call PI2-CMA for "Path Integral Policy Improvement with Covariance Matrix Adaptation". PI2-CMA's main advantage is that it determines the magnitude of the exploration noise automatically.

ICML2012

Cited by in corpus (21)

• End-to-End Training of Deep Visuomotor Policies
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• CEM-RL: Combining evolutionary and gradient-based methods for policy search
• Real-Time Stochastic Optimal Control for Multi-agent Quadrotor Systems
• Shaping Individualized Impedance Landscapes for Gait Training via Reinforcement Learning
• Sample Efficient Path Integral Control under Uncertainty
• Mirror Descent Search and its Acceleration
• Path Integral Guided Policy Search
• Importance mixing: Improving sample reuse in evolutionary policy search methods
• Actor-critic versus direct policy search: a comparison based on sample complexity
• STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulation
• On Entropy Regularized Path Integral Control for Trajectory Optimization
• Variational Policy for Guiding Point Processes
• A Stochastic Differential Equation Framework for Guiding Online User Activities in Closed Loop
• ARES: Adaptive Receding-Horizon Synthesis of Optimal Plans
• Efficient Baseline-free Sampling in Parameter Exploring Policy Gradients: Super Symmetric PGPE
• A Survey of Exploration Methods in Reinforcement Learning
• Supervised Learning and Reinforcement Learning of Feedback Models for Reactive Behaviors: Tactile Feedback Testbed
• Cooperative Path Integral Control for Stochastic Multi-Agent Systems
• FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control
• Guided Policy Improvement for Satisfying STL Tasks using Funnel Adaptation