Derivative-Informed Projected Neural Networks for High-Dimensional Parametric Maps Governed by PDEs

Abstract

Many-query problems, arising from uncertainty quantification, Bayesian inversion, Bayesian optimal experimental design, and optimization under uncertainty-require numerous evaluations of a parameter-to-output map. These evaluations become prohibitive if this parametric map is high-dimensional and involves expensive solution of partial differential equations (PDEs). To tackle this challenge, we propose to construct surrogates for high-dimensional PDE-governed parametric maps in the form of projected neural networks that parsimoniously capture the geometry and intrinsic low-dimensionality of these maps. Specifically, we compute Jacobians of these PDE-based maps, and project the high-dimensional parameters onto a low-dimensional derivative-informed active subspace; we also project the possibly high-dimensional outputs onto their principal subspace. This exploits the fact that many high-dimensional PDE-governed parametric maps can be well-approximated in low-dimensional parameter and output subspace. We use the projection basis vectors in the active subspace as well as the principal output subspace to construct the weights for the first and last layers of the neural network, respectively. This frees us to train the weights in only the low-dimensional layers of the neural network. The architecture of the resulting neural network captures to first order, the low-dimensional structure and geometry of the parametric map. We demonstrate that the proposed projected neural network achieves greater generalization accuracy than a full neural network, especially in the limited training data regime afforded by expensive PDE-based parametric maps. Moreover, we show that the number of degrees of freedom of the inner layers of the projected network is independent of the parameter and output dimensions, and high accuracy can be achieved with weight dimension independent of the discretization dimension.

References in corpus (16)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Geometric deep learning: going beyond Euclidean data
• Solving high-dimensional partial differential equations using deep learning
• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problems
• A Low Effort Approach to Structured CNN Design Using PCA
• Optimal design of acoustic metamaterial cloaks under uncertainty
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• Hessian-based adaptive sparse quadrature for infinite-dimensional Bayesian inverse problems
• Projected Stein Variational Newton: A Fast and Scalable Bayesian Inference Method in High Dimensions
• A-optimal encoding weights for nonlinear inverse problems, with applications to the Helmholtz inverse problem
• Randomized Numerical Linear Algebra: Foundations & Algorithms
• A fast and scalable computational framework for large-scale and high-dimensional Bayesian optimal experimental design
• Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training
• Deep Networks from the Principle of Rate Reduction
• Taylor approximation for chance constrained optimization problems governed by partial differential equations with high-dimensional random parameters

Cited by in corpus (10)

• The Random Feature Model for Input-Output Maps between Banach Spaces
• Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equations
• Residual-based error correction for neural operator accelerated infinite-dimensional Bayesian inverse problems
• Convergence Rates for Learning Linear Operators from Noisy Data
• Parameter Estimation with Dense and Convolutional Neural Networks Applied to the FitzHugh-Nagumo ODE
• Data-driven Uncertainty Quantification in Computational Human Head Models
• Learning the random variables in Monte Carlo simulations with stochastic gradient descent: Machine learning for parametric PDEs and financial derivative pricing
• Optimal Dirichlet Boundary Control by Fourier Neural Operators Applied to Nonlinear Optics
• Nonlinear Level Set Learning for Function Approximation on Sparse Data with Applications to Parametric Differential Equations
• Solving time-dependent parametric PDEs by multiclass classification-based reduced order model