Neural-net-induced Gaussian process regression for function approximation and PDE solution

Abstract

Neural-net-induced Gaussian process (NNGP) regression inherits both the high expressivity of deep neural networks (deep NNs) as well as the uncertainty quantification property of Gaussian processes (GPs). We generalize the current NNGP to first include a larger number of hyperparameters and subsequently train the model by maximum likelihood estimation. Unlike previous works on NNGP that targeted classification, here we apply the generalized NNGP to function approximation and to solving partial differential equations (PDEs). Specifically, we develop an analytical iteration formula to compute the covariance function of GP induced by deep NN with an error-function nonlinearity. We compare the performance of the generalized NNGP for function approximations and PDE solutions with those of GPs and fully-connected NNs. We observe that for smooth functions the generalized NNGP can yield the same order of accuracy with GP, while both NNGP and GP outperform deep NN. For non-smooth functions, the generalized NNGP is superior to GP and comparable or superior to deep NN.

References in corpus (7)

• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations
• Machine Learning of Linear Differential Equations using Gaussian Processes
• Deep Neural Networks as Gaussian Processes
• Inferring solutions of differential equations using noisy multi-fidelity data
• Exponential expressivity in deep neural networks through transient chaos
• Numerical Gaussian Processes for Time-dependent and Non-linear Partial Differential Equations

Cited by in corpus (22)

• B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data
• fPINNs: Fractional Physics-Informed Neural Networks
• Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems
• Deep Learning of Vortex Induced Vibrations
• Multi-fidelity Bayesian Neural Networks: Algorithms and Applications
• Discontinuity Computing using Physics-Informed Neural Network
• Physics-Informed CoKriging: A Gaussian-Process-Regression-Based Multifidelity Method for Data-Model Convergence
• Constraint-Aware Neural Networks for Riemann Problems
• Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations
• Symplectic Neural Networks in Taylor Series Form for Hamiltonian Systems
• Learning elliptic partial differential equations with randomized linear algebra
• Linking Gaussian Process regression with data-driven manifold embeddings for nonlinear data fusion
• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• The Neural Network Approach to Inverse Problems in Differential Equations
• Extreme learning machine collocation for the numerical solution of elliptic PDEs with sharp gradients
• Learning and Meta-Learning of Stochastic Advection-Diffusion-Reaction Systems from Sparse Measurements
• Adaptive Uncertainty-Guided Model Selection for Data-Driven PDE Discovery
• Variational Inference for Nonlinear Inverse Problems via Neural Net Kernels: Comparison to Bayesian Neural Networks, Application to Topology Optimization
• On the Correspondence between Gaussian Processes and Geometric Harmonics
• Fredholm Neural Networks
• Uncertainty Quantification From Scaling Laws in Deep Neural Networks
• Neural Network Gaussian Process Considering Input Uncertainty for Composite Structures Assembly