Progressive Optimal Path Sampling for Closed-Loop Optimal Control Design with Deep Neural Networks

Abstract

Closed-loop optimal control design for high-dimensional nonlinear systems has been a long-standing challenge. Traditional methods, such as solving the associated Hamilton-Jacobi-Bellman equation, suffer from the curse of dimensionality. Recent literature proposed a new promising approach based on supervised learning, by leveraging powerful open-loop optimal control solvers to generate training data and neural networks as efficient high-dimensional function approximators to fit the closed-loop optimal control. This approach successfully handles certain high-dimensional optimal control problems but still performs poorly on more challenging problems. One of the crucial reasons for the failure is the so-called distribution mismatch phenomenon brought by the controlled dynamics. In this paper, we investigate this phenomenon and propose the Progressive Optimal Path Sampling (POPS) method to mitigate this problem. We theoretically prove that this enhanced sampling strategy outperforms both the vanilla approach and the widely used Dataset Aggregation (DAgger) method on the classical linear-quadratic regulator by a factor proportional to the total time duration. We further numerically demonstrate that the proposed sampling strategy significantly improves the performance on tested control problems, including the optimal landing problem of a quadrotor and the optimal reaching problem of a 7 DoF manipulator.

arXiv commentRevised journal version with updated title. Earlier title: Initial Value Problem Enhanced Sampling for Closed-Loop Optimal Control Design with Deep Neural Networks

References in corpus (8)

• Reinforced dynamics for enhanced sampling in large atomic and molecular systems
• Adaptive Deep Learning for High-Dimensional Hamilton-Jacobi-Bellman Equations
• Implicit energy regularization of neural ordinary-differential-equation control
• QRnet: optimal regulator design with LQR-augmented neural networks
• Algorithms of Data Development For Deep Learning and Feedback Design
• Neural Network Optimal Feedback Control with Guaranteed Local Stability
• Offline Supervised Learning V.S. Online Direct Policy Optimization: A Comparative Study and A Unified Training Paradigm for Neural Network-Based Optimal Feedback Control
• Neural network optimal feedback control with enhanced closed loop stability