Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework
arXiv:1912.12970
Abstract
This paper develops a Pontryagin Differentiable Programming (PDP) methodology, which establishes a unified framework to solve a broad class of learning and control tasks. The PDP distinguishes from existing methods by two novel techniques: first, we differentiate through Pontryagin's Maximum Principle, and this allows to obtain the analytical derivative of a trajectory with respect to tunable parameters within an optimal control system, enabling end-to-end learning of dynamics, policies, or/and control objective functions; and second, we propose an auxiliary control system in the backward pass of the PDP framework, and the output of this auxiliary control system is the analytical derivative of the original system's trajectory with respect to the parameters, which can be iteratively solved using standard control tools. We investigate three learning modes of the PDP: inverse reinforcement learning, system identification, and control/planning. We demonstrate the capability of the PDP in each learning mode on different high-dimensional systems, including multi-link robot arm, 6-DoF maneuvering quadrotor, and 6-DoF rocket powered landing.
arXiv commentPublished in NeurIPS 2020, Codes are at https://github.com/wanxinjin/Pontryagin-Differentiable-Programming
References in corpus (26)
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- ADADELTA: An Adaptive Learning Rate Method
- Playing Atari with Deep Reinforcement Learning
- End to End Learning for Self-Driving Cars
- Deep learning for universal linear embeddings of nonlinear dynamics
- Generative Adversarial Imitation Learning
- Learning Continuous Control Policies by Stochastic Value Gradients
- Continuous Deep Q-Learning with Model-based Acceleration
- Path integrals and symmetry breaking for optimal control theory
- Differentiable MPC for End-to-end Planning and Control
- Control of Memory, Active Perception, and Action in Minecraft
- Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control
- Task-based End-to-end Model Learning in Stochastic Optimization
- Deep Learning Approximation for Stochastic Control Problems
- Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning
- Universal Planning Networks
- You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
- Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
- Approximation Methods for Bilevel Programming
- Inverse Optimal Control from Incomplete Trajectory Observations
- An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks
- Composable Planning with Attributes
- Path Integral Networks: End-to-End Differentiable Optimal Control
- MPC-Inspired Neural Network Policies for Sequential Decision Making
- Selection dynamics for deep neural networks
- Variational Integrator Networks for Physically Structured Embeddings