Reliable extrapolation of deep neural operators informed by physics or sparse observations

Abstract

Deep neural operators can learn nonlinear mappings between infinite-dimensional function spaces via deep neural networks. As promising surrogate solvers of partial differential equations (PDEs) for real-time prediction, deep neural operators such as deep operator networks (DeepONets) provide a new simulation paradigm in science and engineering. Pure data-driven neural operators and deep learning models, in general, are usually limited to interpolation scenarios, where new predictions utilize inputs within the support of the training set. However, in the inference stage of real-world applications, the input may lie outside the support, i.e., extrapolation is required, which may result to large errors and unavoidable failure of deep learning models. Here, we address this challenge of extrapolation for deep neural operators. First, we systematically investigate the extrapolation behavior of DeepONets by quantifying the extrapolation complexity via the 2-Wasserstein distance between two function spaces and propose a new behavior of bias-variance trade-off for extrapolation with respect to model capacity. Subsequently, we develop a complete workflow, including extrapolation determination, and we propose five reliable learning methods that guarantee a safe prediction under extrapolation by requiring additional information -- the governing PDEs of the system or sparse new observations. The proposed methods are based on either fine-tuning a pre-trained DeepONet or multifidelity learning. We demonstrate the effectiveness of the proposed framework for various types of parametric PDEs. Our systematic comparisons provide practical guidelines for selecting a proper extrapolation method depending on the available information, desired accuracy, and required inference speed.

References in corpus (24)

• Overcoming catastrophic forgetting in neural networks
• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• Gaussian Error Linear Units (GELUs)
• DeepXDE: A deep learning library for solving differential equations
• Reconciling modern machine learning practice and the bias-variance trade-off
• Fourier Neural Operator for Parametric Partial Differential Equations
• A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems
• A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks
• Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons
• A physics-informed variational DeepONet for predicting the crack path in brittle materials
• DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks
• Operator learning for predicting multiscale bubble growth dynamics
• Interfacing Finite Elements with Deep Neural Operators for Fast Multiscale Modeling of Mechanics Problems
• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
• A physics-informed operator regression framework for extracting data-driven continuum models
• Learning Operators with Coupled Attention
• Nonlocal Kernel Network (NKN): a Stable and Resolution-Independent Deep Neural Network
• Scalable Uncertainty Quantification for Deep Operator Networks using Randomized Priors
• DeepONet prediction of linear instability waves in high-speed boundary layers
• Generic bounds on the approximation error for physics-informed (and) operator learning
• Multiscale DeepONet for Nonlinear Operators in Oscillatory Function Spaces for Building Seismic Wave Responses
• Accelerated replica exchange stochastic gradient Langevin diffusion enhanced Bayesian DeepONet for solving noisy parametric PDEs
• On the Activation Function Dependence of the Spectral Bias of Neural Networks
• Neural and spectral operator surrogates: unified construction and expression rate bounds

Cited by in corpus (14)

• Data-Driven Design for Metamaterials and Multiscale Systems: A Review
• A multi-fidelity deep operator network (DeepONet) for fusing simulation and monitoring data: Application to real-time settlement prediction during tunnel construction
• In-Context Operator Learning with Data Prompts for Differential Equation Problems
• Improved generalization with deep neural operators for engineering systems: Path towards digital twin
• A Mathematical Guide to Operator Learning
• Deep Neural Operator Enabled Concurrent Multitask Design for Multifunctional Metamaterials under Heterogeneous Fields
• An extrapolation-driven network architecture for physics-informed deep learning
• Scattering with Neural Operators
• Local neural operator for solving transient partial differential equations on varied domains
• Fold Bifurcation Identification through Scientific Machine Learning
• Neural Green's Operators for Parametric Partial Differential Equations
• Fast meta-solvers for 3D complex-shape scatterers using neural operators trained on a non-scattering problem
• Hamiltonian Learning via Inverse Physics-Informed Neural Networks
• Learning Post-Newtonian Corrections from Numerical Relativity