Offline Supervised Learning V.S. Online Direct Policy Optimization: A Comparative Study and A Unified Training Paradigm for Neural Network-Based Optimal Feedback Control
arXiv:2211.15930 · doi:10.1016/j.physd.2024.134130
Abstract
This work is concerned with solving neural network-based feedback controllers efficiently for optimal control problems. We first conduct a comparative study of two prevalent approaches: offline supervised learning and online direct policy optimization. Albeit the training part of the supervised learning approach is relatively easy, the success of the method heavily depends on the optimal control dataset generated by open-loop optimal control solvers. In contrast, direct policy optimization turns the optimal control problem into an optimization problem directly without any requirement of pre-computing, but the dynamics-related objective can be hard to optimize when the problem is complicated. Our results underscore the superiority of offline supervised learning in terms of both optimality and training time. To overcome the main challenges, dataset and optimization, in the two approaches respectively, we complement them and propose the Pre-train and Fine-tune strategy as a unified training paradigm for optimal feedback control, which further improves the performance and robustness significantly. Our code is accessible at https://github.com/yzhao98/DeepOptimalControl.
arXiv commentAccepted for publication in Physica D
References in corpus (8)
- Learning robust perceptive locomotion for quadrupedal robots in the wild
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- An Algorithmic Perspective on Imitation Learning
- Deep Learning Approximation for Stochastic Control Problems
- Adaptive Deep Learning for High-Dimensional Hamilton-Jacobi-Bellman Equations
- Implicit energy regularization of neural ordinary-differential-equation control
- A Neural Network Approach for High-Dimensional Optimal Control Applied to Multi-Agent Path Finding
- Optimal Feedback Law Recovery by Gradient-Augmented Sparse Polynomial Regression