Optimizing Variational Physics-Informed Neural Networks Using Least Squares

Abstract

Variational Physics-Informed Neural Networks often suffer from poor convergence when using stochastic gradient-descent-based optimizers. By introducing a Least Squares solver for the weights of the last layer of the neural network, we improve the convergence of the loss during training in most practical scenarios. This work analyzes the computational cost of the resulting hybrid Least-Squares/Gradient-Descent optimizer and explains how to implement it efficiently. In particular, we show that a traditional implementation based on backward-mode automatic differentiation leads to a prohibitively expensive algorithm. To remedy this, we propose using either forward-mode automatic differentiation or an ultraweak-type scheme that avoids the differentiation of trial functions in the discrete weak formulation. The proposed alternatives are up to one hundred times faster than the traditional one, recovering a computational cost-per-iteration similar to that of a conventional gradient-descent-based optimizer alone. To support our analysis, we derive computational estimates and conduct numerical experiments in one- and two-dimensional problems.

Published work

References in corpus (9)

• Solving high-dimensional partial differential equations using deep learning
• hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition
• Weak Adversarial Networks for High-dimensional Partial Differential Equations
• A Review of automatic differentiation and its efficient implementation
• Solving parametric PDE problems with artificial neural networks
• Enforcing Dirichlet boundary conditions in physics-informed neural networks and variational physics-informed neural networks
• Neural Networks to solve Partial Differential Equations: a Comparison with Finite Elements
• Neural Control of Discrete Weak Formulations: Galerkin, Least-Squares and Minimal-Residual Methods with Quasi-Optimal Weights
• Learning quantities of interest from parametric PDEs: An efficient neural-weighted Minimal Residual approach

Cited by in corpus (2)

• An $r$-adaptive finite element method using neural networks for parametric self-adjoint elliptic problem
• Neural network methods for Neumann series problems of Perron-Frobenius operators