Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimization

Abstract

In this work, we investigate the use of data-driven equation discovery for dynamical systems to model and forecast continuous-time dynamics of unconstrained optimization problems. To avoid expensive evaluations of the objective function and its gradient, we leverage trajectory data on the optimization variables to learn the continuous-time dynamics associated with gradient descent, Newton's method, and ADAM optimization. The discovered gradient flows are then solved as a surrogate for the original optimization problem. To this end, we introduce the Learned Gradient Flow (LGF) optimizer, which is equipped to build surrogate models of variable polynomial order in full- or reduced-dimensional spaces at user-defined intervals in the optimization process. We demonstrate the efficacy of this approach on several standard problems from engineering mechanics and scientific machine learning, including two inverse problems, structural topology optimization, and two forward solves with different discretizations. Our results suggest that the learned gradient flows can significantly expedite convergence by capturing critical features of the optimization trajectory while avoiding expensive evaluations of the objective and its gradient.

44 pages, 13 figures. Submitted to CMAME. Changed Topology Optimization example to be 250% acceleration

References in corpus (13)

• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• Discovering governing equations from data: Sparse identification of nonlinear dynamical systems
• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• Machine Learning of Linear Differential Equations using Gaussian Processes
• Derivative-free optimization methods
• PySINDy: A comprehensive Python package for robust sparse system identification
• Using Noisy or Incomplete Data to Discover Models of Spatiotemporal Dynamics
• Fixed-Time Stable Gradient Flows: Applications to Continuous-Time Optimization
• Derivative-based SINDy (DSINDy): Addressing the challenge of discovering governing equations from noisy data
• Analysis of Stochastic Gradient Descent in Continuous Time
• Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation
• Gradient flows and proximal splitting methods: A unified view on accelerated and stochastic optimization
• Accelerating the convergence of Newton's method for nonlinear elliptic PDEs using Fourier neural operators