Differentiable MPC for End-to-end Planning and Control

Abstract

We present foundations for using Model Predictive Control (MPC) as a differentiable policy class for reinforcement learning in continuous state and action spaces. This provides one way of leveraging and combining the advantages of model-free and model-based approaches. Specifically, we differentiate through MPC by using the KKT conditions of the convex approximation at a fixed point of the controller. Using this strategy, we are able to learn the cost and dynamics of a controller via end-to-end learning. Our experiments focus on imitation learning in the pendulum and cartpole domains, where we learn the cost and dynamics terms of an MPC policy class. We show that our MPC policies are significantly more data-efficient than a generic neural network and that our method is superior to traditional system identification in a setting where the expert is unrealizable.

arXiv commentNeurIPS 2018

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Optimization-Based Autonomous Racing of 1:43 Scale RC Cars
• Learning Continuous Control Policies by Stochastic Value Gradients
• Continuous Deep Q-Learning with Model-based Acceleration
• Control of Memory, Active Perception, and Action in Minecraft
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• The Predictron: End-To-End Learning and Planning
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Universal Planning Networks
• Learning model-based planning from scratch
• Path Integral Networks: End-to-End Differentiable Optimal Control
• MPC-Inspired Neural Network Policies for Sequential Decision Making
• MBMF: Model-Based Priors for Model-Free Reinforcement Learning
• A Fast Integrated Planning and Control Framework for Autonomous Driving via Imitation Learning

Cited by in corpus (48)

• Safe Reinforcement Learning Using Robust MPC
• Autonomous Drone Racing: A Survey
• Implicit Graph Neural Networks
• Deep Imitative Models for Flexible Inference, Planning, and Control
• Adaptive Force-Based Control of Dynamic Legged Locomotion over Uneven Terrain
• Neural Ordinary Differential Equation Control of Dynamics on Graphs
• PyPose: A Library for Robot Learning with Physics-based Optimization
• Learning Deep Stochastic Optimal Control Policies using Forward-Backward SDEs
• Learning Robotic Navigation from Experience: Principles, Methods, and Recent Results
• Objective Mismatch in Model-based Reinforcement Learning
• DiffTune-MPC: Closed-Loop Learning for Model Predictive Control
• On the model-based stochastic value gradient for continuous reinforcement learning
• Improving Robot Dual-System Motor Learning with Intrinsically Motivated Meta-Control and Latent-Space Experience Imagination
• Learning to Optimize in Model Predictive Control
• Neural Lyapunov Differentiable Predictive Control
• Monotone operator equilibrium networks
• Gradient-Aware Model-based Policy Search
• Robust-Adaptive Control of Linear Systems: beyond Quadratic Costs
• Differentiable Moving Horizon Estimation for Robust Flight Control
• Latent Representation in Human-Robot Interaction with Explicit Consideration of Periodic Dynamics
• End-to-End Reinforcement Learning of Koopman Models for Economic Nonlinear Model Predictive Control
• Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework
• The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning
• The Differentiable Cross-Entropy Method
• Towards Learning Affine-Invariant Representations via Data-Efficient CNNs
• Deep Kinematic Models for Kinematically Feasible Vehicle Trajectory Predictions
• Neural Weighted A*: Learning Graph Costs and Heuristics with Differentiable Anytime A*
• SENSORIMOTOR GRAPH: Action-Conditioned Graph Neural Network for Learning Robotic Soft Hand Dynamics
• Expert-Calibrated Learning for Online Optimization with Switching Costs
• NOVAS: Non-convex Optimization via Adaptive Stochastic Search for End-to-End Learning and Control
• Safety Reinforced Model Predictive Control (SRMPC): Improving MPC with Reinforcement Learning for Motion Planning in Autonomous Driving
• A view on learning robust goal-conditioned value functions: Interplay between RL and MPC
• Neural Lyapunov Model Predictive Control: Learning Safe Global Controllers from Sub-optimal Examples
• Differentiable Implicit Layers
• Deep Learning Alternative to Explicit Model Predictive Control for Unknown Nonlinear Systems
• Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles
• Proximal Mapping for Deep Regularization
• TOMA: Topological Map Abstraction for Reinforcement Learning
• Robust Implicit Networks via Non-Euclidean Contractions
• Learning MDPs from Features: Predict-Then-Optimize for Sequential Decision Problems by Reinforcement Learning
• Task-aware Privacy Preservation for Multi-dimensional Data
• A novel approach to model exploration for value function learning
• Sufficiently Accurate Model Learning
• Improving Hyperparameter Optimization by Planning Ahead
• Generative Temporal Difference Learning for Infinite-Horizon Prediction
• Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning
• Differentiable Robust LQR Layers
• Neural Algorithmic Reasoners are Implicit Planners