Learning to Optimize in Model Predictive Control

Abstract

Sampling-based Model Predictive Control (MPC) is a flexible control framework that can reason about non-smooth dynamics and cost functions. Recently, significant work has focused on the use of machine learning to improve the performance of MPC, often through learning or fine-tuning the dynamics or cost function. In contrast, we focus on learning to optimize more effectively. In other words, to improve the update rule within MPC. We show that this can be particularly useful in sampling-based MPC, where we often wish to minimize the number of samples for computational reasons. Unfortunately, the cost of computational efficiency is a reduction in performance; fewer samples results in noisier updates. We show that we can contend with this noise by learning how to update the control distribution more effectively and make better use of the few samples that we have. Our learned controllers are trained via imitation learning to mimic an expert which has access to substantially more samples. We test the efficacy of our approach on multiple simulated robotics tasks in sample-constrained regimes and demonstrate that our approach can outperform a MPC controller with the same number of samples.

arXiv commentProceedings of the IEEE Conference on Robotics and Automation (ICRA), 2022. Paper is 6 pages with 2 figures and 2 tables

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• On the Convergence of Adam and Beyond
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
• Neural Optimizer Search with Reinforcement Learning
• Differentiable MPC for End-to-end Planning and Control
• Learning to Learn without Gradient Descent by Gradient Descent
• Learning to Optimize: A Primer and A Benchmark
• Learning to Optimize Neural Nets
• Exploring Model-based Planning with Policy Networks
• Learning Gradient Descent: Better Generalization and Longer Horizons
• Path Integral Networks: End-to-End Differentiable Optimal Control
• MPC-Inspired Neural Network Policies for Sequential Decision Making
• Acceleration of Gradient-based Path Integral Method for Efficient Optimal and Inverse Optimal Control
• Unsupervised Visuomotor Control through Distributional Planning Networks
• Information Theoretic Model Predictive Q-Learning